Building an LLM application is one thing. Keeping it reliable after every change to code, prompts, models, or dependencies is another. This is why I think CI/CD for LLM applications is becoming an important skill for AI engineers.
An LLM application needs the same engineering discipline as a normal software system, but with an additional challenge: model outputs are probabilistic. Traditional unit tests alone cannot tell you whether an AI application still behaves correctly. Modern evaluation guidance recommends combining automated tests with task-specific evals and continuously evaluating changes.
In this tutorial, I’ll build a simple production-style pipeline using GitHub Actions, Docker, Python, automated tests, and local LLM evaluations. We’ll keep the stack completely free by using a local model instead of a paid API.
CI/CD Pipeline for LLM Applications
The pipeline will follow this flow:
The important idea is that deployment should happen only after the application passes both software tests and AI-specific evaluations.
Step 1: Create the LLM Application
Let’s assume we have a simple Python application:
llm-app/
│
├── app/
│ ├── init.py
│ └── llm_service.py
│
├── tests/
│ ├── test_unit.py
│ └── test_integration.py
│
├── evals/
│ └── test_cases.json
│
├── requirements.txt
├── Dockerfile
└── .github/
└── workflows/
└── ci-cd.yml
For the model, we’ll use a local Ollama installation during development:
ollama pull qwen3
A simple LLM service could look like this:
from ollama import chat
def generate_response(prompt):
response = chat(
model="qwen3",
messages=[
{
"role": "user",
"content": prompt
}
]
)
return response.message.contentThis keeps the application simple enough to focus on the CI/CD architecture.
Step 2: Add Traditional Unit Tests
The first layer of our pipeline should be normal software testing. For example:
def test_basic_math():
result = 2 + 2
assert result == 4Run the tests locally:
pytest
This catches ordinary programming problems such as broken imports, incorrect functions, and unexpected application behavior.
But there is a problem. A unit test doesn’t tell us whether the LLM produced a useful answer. That’s where evals come in.
Step 3: Add LLM Evaluations
LLM applications are nondeterministic, so testing only whether a function executes successfully isn’t enough. Task-specific evaluations allow you to measure whether the application is actually producing the behavior you want.
Create a small evaluation dataset:
[
{
"input": "Explain machine learning in one sentence.",
"expected_keyword": "data"
},
{
"input": "What is Python?",
"expected_keyword": "programming"
}
]Then create an evaluation script:
import json
from app.llm_service import generate_response
with open("evals/test_cases.json") as f:
test_cases = json.load(f)
def run_evals():
passed = 0
for case in test_cases:
response = generate_response(case["input"])
keyword = case["expected_keyword"].lower()
if keyword in response.lower():
passed += 1
score = passed / len(test_cases)
print(f"Eval score: {score:.2f}")
return score
if __name__ == "__main__":
score = run_evals()
if score < 0.8:
raise SystemExit("LLM evaluation failed")This is deliberately simple. In a real application, your evals could measure things such as factual correctness, retrieval quality, structured output validity, tool selection, or task completion.
The important principle is to define what good behavior means before deploying changes.
Preparing for AI/ML Interviews?
If you’re building projects like this while preparing for AI/ML interviews, I’d recommend Cracking Your First AI/ML Interview. I put it together to help you turn your AI/ML knowledge and projects into interview-ready skills.
Step 4: Containerize the Application
Now let’s package the application using Docker:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "-m", "app"]Build it locally:
docker build -t llm-app:latest .
Docker gives us a reproducible environment, which is particularly useful when moving an AI application from development to staging or production.
Step 5: Add GitHub Actions
Now we automate everything. Create:
.github/workflows/ci-cd.yml
Then:
name: LLM Application CI/CD
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run unit tests
run: pytest tests/test_unit.py
- name: Run integration tests
run: pytest tests/test_integration.pyNow every pull request can automatically run your software tests.
Step 6: Run LLM Evals in CI
Add another step:
- name: Run LLM evaluations
run: python evals/run_evals.pyIf the eval score falls below your threshold, the workflow should stop. This is one of the biggest differences between traditional CI/CD and LLM application CI/CD.
Step 7: Manage Environment Variables and Secrets
Never hard-code API keys, database passwords, tokens, or production credentials inside your repository. GitHub Actions supports repository and environment secrets, and environments can also add approval rules before production jobs proceed.
For example:
env:
MODEL_NAME: qwen3For sensitive values:
env:
API_KEY: ${{ secrets.API_KEY }}For production, I would create separate environments:
development
staging
production
GitHub environments can restrict which branches deploy and can require manual approval before a production job proceeds.
Step 8: Build and Deploy
Once everything passes, build the Docker image:
build:
needs: test
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build Docker image
run: |
docker build \
-t llm-app:${{ github.sha }} .Notice that we’re tagging the image with the Git commit SHA. This makes deployments traceable.
The Takeaway
When I think about the AI engineering skills that matter today, CI/CD is easy to overlook. It’s tempting to focus entirely on models, prompts, RAG, and agents. But once an AI application reaches production, engineering discipline becomes just as important.
A good AI engineer should be able to take an LLM application from:
That’s the mindset I would develop if you’re preparing for AI engineering roles. You don’t need expensive infrastructure to learn it. Start with Python, GitHub Actions, Docker, pytest, a local LLM, and a small evaluation dataset.
I hope you liked this article on how to set up CI/CD for LLM applications. For more tips on AI and machine learning, you can follow me on Instagram.
