100 CI/CD for AI Apps resources for developers

Run test cases against your prompts using a matrix of LLM providers. Integrates into GitHub Actions to fail builds if semantic similarity or assertion checks fail.

A Python framework for unit testing LLM outputs. It uses 'LLM-as-a-judge' to score metrics like faithfulness, relevancy, and hallucination during the CI phase.

An open-source library for detecting vulnerabilities in LLMs, including prompt injection and biases. Use it in CI to scan for regressions in model behavior.

Specific metrics for Retrieval Augmented Generation. Automate the evaluation of context precision and recall within your CI pipeline to ensure retrieval quality.

Use LangSmith to run evaluation datasets automatically on every commit. It provides a dashboard to compare performance across different versions of your application.

A high-performance framework for running evals in parallel. Use their CLI to trigger evaluation runs that track performance improvements over time.

Provides 'feedback functions' to evaluate LLM apps. Can be integrated into build steps to verify that new prompt versions meet quality thresholds.

A platform for automated LLM monitoring and evaluation. Useful for setting up CI gates that prevent the deployment of models with high error rates.

Open-source toolkit for modular evaluation of RAG pipelines. Ideal for testing specific components like the retriever or the generator in isolation.

Provides automated checks for LLM responses. Use it to ensure that your model's output adheres to specific guidelines or formatting requirements in CI.

An open-source prompt management platform. It allows you to decouple prompts from code, enabling instant updates without redeploying the entire application.

Version and manage prompts in a centralized registry. Use the SDK to pull specific prompt versions in production while maintaining a history in Git.

Manage large model files and datasets alongside your code. Use DVC to version the data used for fine-tuning or few-shot prompting in your CI pipeline.

Track prompt iterations and visualize results. Integrate with CI to log the performance of each prompt variant during automated testing.

A tool for prompt engineering that includes versioning and evaluation. It helps bridge the gap between prompt design and CI/CD deployment workflows.

A central repository for sharing and versioning LangChain prompts. Use it to pull vetted prompt templates into your build process.

An evaluation and observability platform. Use their CI integration to run regression tests on prompts before they are merged into the main branch.

Use Git Large File Storage to manage local model weights or configuration binaries within your repository, ensuring they are synced across CI environments.

A middleware for logging and managing prompts. Use it to version-tag prompts in CI to ensure the correct version is used by the application at runtime.

Standardize prompts as YAML files in your repo. This allows for standard Git diffs and automated validation of variable placeholders in CI scripts.

Streamlines the deployment of AI-powered edge functions. Provides built-in support for streaming responses and automated preview deployments.

A framework for building and deploying machine learning services. Use it to package models as Docker containers for automated deployment via CI/CD.

Run LLMs on any cloud provider with optimized cost. Use SkyPilot in CI to spin up GPU instances for heavy evaluation jobs or model fine-tuning.

Deploy Dockerized AI applications to bare-metal GPU servers with zero-downtime. Excellent for self-hosting models like Llama 3 using CI pipelines.

Deploy AI models directly to the edge. Use the Wrangler CLI in your CI/CD pipeline to automate the deployment of inference functions.

An open-source model packaging tool. Use it in CI to ensure your model environment is reproducible and ready for production serving.

Standardizes the environment for ML models. Use Cog in your build step to create a production-ready Docker container with all GPU dependencies.

Automate the provisioning of Pinecone or Milvus indexes. Ensure your vector database schema and metadata filtering are version-controlled.

Easily deploy GPU-backed containers. Use Fly's GitHub Action to automate the deployment of inference servers with attached local storage.

Implement GitOps for AI models. Use ArgoCD to manage the state of model deployments on Kubernetes, ensuring the live version matches the repo.