DEVOPS TRIBUNE ENGINEERING INTELLIGENCE FOR THE REAL WORLD

Local LLMs for DevOps: Running Offline AI with Ollama & vLLM for Platform Automation

How to deploy open-weights language models inside private VPC boundaries: hardware sizing, Ollama vs vLLM inference engines, Kubernetes GPU operators, and automated incident triage scripts.

1. The Infrastructure Data Sovereignty & Compliance Imperative

While public generative AI APIs (such as OpenAI, Anthropic, or Google Gemini) offer frontier reasoning capabilities, integrating them directly into enterprise infrastructure pipelines introduces severe compliance and security violations:

The Local LLM Rule of Thumb

Use frontier commercial APIs for high-level exploratory architectural synthesis where zero confidential code is passed. Use self-hosted, private local models (Ollama / vLLM) for high-frequency CI/CD automation, log summarization, Terraform policy validation, and incident triage runbooks.

2. Model Selection & Hardware VRAM Sizing Matrix

Modern open-weights models in the 7B to 14B parameter range deliver specialized coding and reasoning capabilities that rival larger models on structured DevOps tasks while fitting comfortably on cost-effective enterprise GPUs (such as NVIDIA L4 24GB or A10G 24GB) or Apple Silicon workstations.

Model Parameters Quantization VRAM Required Ideal DevOps Workload
Qwen 2.5 Coder 7B Q4_K_M (GGUF) ~5.5 GB VRAM Fast syntax checking, Dockerfile optimization, and simple shell script generation on local developer laptops.
DeepSeek-R1 Distill Qwen 7B / 14B Q8_0 / AWQ ~9.5 GB / ~16 GB Chain-of-thought root-cause analysis (RCA) and complex incident log correlation where step-by-step reasoning is required.
Llama 3.3 8B FP16 / AWQ ~16 GB VRAM General platform engineering assistant, pull request summaries, and multi-cloud Terraform translation.
Mistral Nemo 12B Q4_K_M ~9 GB VRAM Large 128k context window processing for auditing full monorepo Helm charts and multi-service compose stacks.

3. Runtime Architecture: Ollama vs vLLM

Choosing the correct inference engine depends entirely on whether the workload serves an individual developer or an entire platform cluster:

Dimension Ollama (llama.cpp engine) vLLM (PagedAttention engine)
Primary Target Developer workstations, single-node bastions, local CLI scripts. Production multi-tenant microservices, Kubernetes clusters, shared API gateways.
Memory Management Standard GGUF memory mapping; sequential execution under concurrent requests. PagedAttention: Near-zero memory waste, dynamic KV cache allocation, continuous batching.
Concurrency & Throughput Low: Designed for 1–2 concurrent users; queues subsequent requests. High: Handles dozens of parallel inference streams per GPU with minimal latency penalty.
API Compatibility Native REST API (/api/generate) plus basic OpenAI-compatible endpoints. Drop-in replacement for OpenAI API (/v1/chat/completions, /v1/embeddings).

4. Production Deployment: Docker Compose & Kubernetes

Recipe A: Local Workstation / Bastion Deployment (Ollama)

Run Ollama as an isolated container service with persistent model caching and automatic GPU acceleration:

compose.yaml (Ollama GPU) Docker Compose v2
services:
  ollama:
    image: ollama/ollama:latest
    container_name: local-ai
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434" # Bound strictly to localhost for security
    volumes:
      - ollama_models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama_models:

Recipe B: Enterprise Cluster Deployment (vLLM on Kubernetes)

Deploy vLLM into a dedicated Kubernetes cluster using the NVIDIA GPU Operator:

vllm-deployment.yaml Kubernetes Manifest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-inference-engine
  namespace: ai-platform
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-engine
  template:
    metadata:
      labels:
        app: vllm-engine
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.6.2
          args:
            - "--model"
            - "Qwen/Qwen2.5-Coder-7B-Instruct"
            - "--max-model-len"
            - "8192"
            - "--gpu-memory-utilization"
            - "0.90"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 32Gi
              cpu: 4
            requests:
              nvidia.com/gpu: 1
              memory: 16Gi
              cpu: 2
          volumeMounts:
            - name: model-cache
              mountPath: /root/.cache/huggingface
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: ai-model-weights-pvc

5. Practical Platform Engineering Automation Workflows

Automated Kubernetes Pod Triage Script

Pipe previous container crash logs directly into a local Ollama model to generate an immediate root cause assessment:

triage-pod.py Python Automation
import subprocess
import requests
import json
import sys

def triage_failing_pod(pod_name, namespace="production"):
    # 1. Extract previous crash logs and termination events
    logs = subprocess.check_output(
        ["kubectl", "logs", pod_name, "-n", namespace, "--previous", "--tail=80"],
        text=True
    )
    
    events = subprocess.check_output(
        ["kubectl", "get", "events", "-n", namespace, 
         f"--field-selector=involvedObject.name={pod_name}"],
        text=True
    )

    prompt = f"""
You are a senior Kubernetes SRE. Analyze the following crash logs and pod events:

--- POD EVENTS ---
{events}

--- CRASH LOGS (stderr) ---
{logs}

Provide:
1. Exact Root Cause (1 sentence)
2. Culprit Component (e.g. JVM heap, missing env var, DB timeout)
3. Immediate Remediation Command
"""

    # 2. Query local offline Ollama instance
    response = requests.post(
        "http://127.0.0.1:11434/api/generate",
        json={
            "model": "qwen2.5-coder:7b",
            "prompt": prompt,
            "stream": False
        }
    )
    
    print("\n=== AI TRIAGE ASSESSMENT ===")
    print(response.json().get("response"))

if __name__ == "__main__":
    triage_failing_pod(sys.argv[1])

6. Operational Guardrails & Hallucination Defenses

Language models must never be granted direct, unsupervised write access to production cloud environments. Implement these operational boundaries: