Local LLMs for DevOps: Running Offline AI with Ollama & vLLM for Platform Automation
How to deploy open-weights language models inside private VPC boundaries: hardware sizing, Ollama vs vLLM inference engines, Kubernetes GPU operators, and automated incident triage scripts.
1. The Infrastructure Data Sovereignty & Compliance Imperative
While public generative AI APIs (such as OpenAI, Anthropic, or Google Gemini) offer frontier reasoning capabilities, integrating them directly into enterprise infrastructure pipelines introduces severe compliance and security violations:
- Credential & Secret Exposure: Application crash dumps, Kubernetes manifests, and Terraform plan files frequently contain accidentally masked credentials, database connection strings, and private internal DNS topologies. Streaming these logs to third-party endpoints violates SOC2, HIPAA, and ISO 27001 data residency controls.
- Air-Gapped & Private VPC Isolation: Critical production bastion hosts, edge Kubernetes clusters, and sovereign government enclaves often have strict egress firewalls that prohibit outbound internet connections to public AI APIs.
- Predictable OpEx vs Runaway Token Billing: Feeding gigabytes of observability traces and CI/CD logs into commercial LLMs incurs exponential token billing spikes. Self-hosting fixed-capacity GPU or CPU inference nodes establishes a deterministic monthly operational cost.
Use frontier commercial APIs for high-level exploratory architectural synthesis where zero confidential code is passed. Use self-hosted, private local models (Ollama / vLLM) for high-frequency CI/CD automation, log summarization, Terraform policy validation, and incident triage runbooks.
2. Model Selection & Hardware VRAM Sizing Matrix
Modern open-weights models in the 7B to 14B parameter range deliver specialized coding and reasoning capabilities that rival larger models on structured DevOps tasks while fitting comfortably on cost-effective enterprise GPUs (such as NVIDIA L4 24GB or A10G 24GB) or Apple Silicon workstations.
| Model | Parameters | Quantization | VRAM Required | Ideal DevOps Workload |
|---|---|---|---|---|
| Qwen 2.5 Coder | 7B | Q4_K_M (GGUF) | ~5.5 GB VRAM | Fast syntax checking, Dockerfile optimization, and simple shell script generation on local developer laptops. |
| DeepSeek-R1 Distill Qwen | 7B / 14B | Q8_0 / AWQ | ~9.5 GB / ~16 GB | Chain-of-thought root-cause analysis (RCA) and complex incident log correlation where step-by-step reasoning is required. |
| Llama 3.3 | 8B | FP16 / AWQ | ~16 GB VRAM | General platform engineering assistant, pull request summaries, and multi-cloud Terraform translation. |
| Mistral Nemo | 12B | Q4_K_M | ~9 GB VRAM | Large 128k context window processing for auditing full monorepo Helm charts and multi-service compose stacks. |
3. Runtime Architecture: Ollama vs vLLM
Choosing the correct inference engine depends entirely on whether the workload serves an individual developer or an entire platform cluster:
| Dimension | Ollama (llama.cpp engine) | vLLM (PagedAttention engine) |
|---|---|---|
| Primary Target | Developer workstations, single-node bastions, local CLI scripts. | Production multi-tenant microservices, Kubernetes clusters, shared API gateways. |
| Memory Management | Standard GGUF memory mapping; sequential execution under concurrent requests. | PagedAttention: Near-zero memory waste, dynamic KV cache allocation, continuous batching. |
| Concurrency & Throughput | Low: Designed for 1–2 concurrent users; queues subsequent requests. | High: Handles dozens of parallel inference streams per GPU with minimal latency penalty. |
| API Compatibility | Native REST API (/api/generate) plus basic OpenAI-compatible endpoints. |
Drop-in replacement for OpenAI API (/v1/chat/completions, /v1/embeddings). |
4. Production Deployment: Docker Compose & Kubernetes
Recipe A: Local Workstation / Bastion Deployment (Ollama)
Run Ollama as an isolated container service with persistent model caching and automatic GPU acceleration:
services:
ollama:
image: ollama/ollama:latest
container_name: local-ai
restart: unless-stopped
ports:
- "127.0.0.1:11434:11434" # Bound strictly to localhost for security
volumes:
- ollama_models:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
ollama_models:
Recipe B: Enterprise Cluster Deployment (vLLM on Kubernetes)
Deploy vLLM into a dedicated Kubernetes cluster using the NVIDIA GPU Operator:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-inference-engine
namespace: ai-platform
spec:
replicas: 1
selector:
matchLabels:
app: vllm-engine
template:
metadata:
labels:
app: vllm-engine
spec:
containers:
- name: vllm
image: vllm/vllm-openai:v0.6.2
args:
- "--model"
- "Qwen/Qwen2.5-Coder-7B-Instruct"
- "--max-model-len"
- "8192"
- "--gpu-memory-utilization"
- "0.90"
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: 1
memory: 32Gi
cpu: 4
requests:
nvidia.com/gpu: 1
memory: 16Gi
cpu: 2
volumeMounts:
- name: model-cache
mountPath: /root/.cache/huggingface
volumes:
- name: model-cache
persistentVolumeClaim:
claimName: ai-model-weights-pvc
5. Practical Platform Engineering Automation Workflows
Automated Kubernetes Pod Triage Script
Pipe previous container crash logs directly into a local Ollama model to generate an immediate root cause assessment:
import subprocess
import requests
import json
import sys
def triage_failing_pod(pod_name, namespace="production"):
# 1. Extract previous crash logs and termination events
logs = subprocess.check_output(
["kubectl", "logs", pod_name, "-n", namespace, "--previous", "--tail=80"],
text=True
)
events = subprocess.check_output(
["kubectl", "get", "events", "-n", namespace,
f"--field-selector=involvedObject.name={pod_name}"],
text=True
)
prompt = f"""
You are a senior Kubernetes SRE. Analyze the following crash logs and pod events:
--- POD EVENTS ---
{events}
--- CRASH LOGS (stderr) ---
{logs}
Provide:
1. Exact Root Cause (1 sentence)
2. Culprit Component (e.g. JVM heap, missing env var, DB timeout)
3. Immediate Remediation Command
"""
# 2. Query local offline Ollama instance
response = requests.post(
"http://127.0.0.1:11434/api/generate",
json={
"model": "qwen2.5-coder:7b",
"prompt": prompt,
"stream": False
}
)
print("\n=== AI TRIAGE ASSESSMENT ===")
print(response.json().get("response"))
if __name__ == "__main__":
triage_failing_pod(sys.argv[1])
6. Operational Guardrails & Hallucination Defenses
Language models must never be granted direct, unsupervised write access to production cloud environments. Implement these operational boundaries:
- Read-Only Ingestion: The model may ingest logs, configurations, and telemetry, but all remediation actions must be output as suggested commands or pull requests requiring human review.
- Deterministic Policy Gates: Never use an LLM as a replacement for deterministic security scanners (such as Trivy, Checkov, or OPA/Gatekeeper). The LLM explains and summarizes findings; deterministic linters enforce the policy.
- Structured JSON Output Enforcement: Use constrained decoding (e.g. Ollama's
format: "json"schema parameter) to guarantee that automation scripts receive valid, machine-parseable responses without unpredictable conversational preamble.