Deploying an LLM Gateway on Kubernetes with Helm

Running an LLM gateway in production means it needs to be reliable, scalable, and manageable. Kubernetes gives you all three — but setting up a gateway with its database, cache, and secrets correctly takes some planning. DeltaLLM ships a Helm chart with three pre-built profiles that cover evaluation, standard production, and high-availability deployments.

Three Deployment Profiles

Profile	Best for	What's included
Eval	Quick testing, demos	Bundled PostgreSQL + Redis. One replica. Everything self-contained.
Production	Standard workloads	External PostgreSQL + Redis. Proper secret management. Ready for real traffic.
High Availability	Mission-critical deployments	3+ replicas, HPA, PDB, topology spread, ServiceMonitor.

Quick Start: Evaluation Install

Get a working gateway in under two minutes. This bundles PostgreSQL and Redis inside the cluster — no external dependencies.

terminal

# Add the Helm repo
helm repo add deltallm https://deltawi.github.io/deltallm
helm repo update

# Generate required secrets
export DELTALLM_MASTER_KEY="$(python3 -c 'import secrets; print("sk-" + secrets.token_hex(20) + "A1")')"
export DELTALLM_SALT_KEY="$(openssl rand -hex 32)"

# Install with the eval profile
helm install deltallm deltallm/deltallm \
  --namespace deltallm \
  --create-namespace \
  -f https://deltawi.github.io/deltallm/values-eval-<version>.yaml \
  --set secret.values.masterKey="$DELTALLM_MASTER_KEY" \
  --set secret.values.saltKey="$DELTALLM_SALT_KEY" \
  --set-string env[0].name=PLATFORM_BOOTSTRAP_ADMIN_EMAIL \
  --set-string env[0][email protected] \
  --set-string env[1].name=PLATFORM_BOOTSTRAP_ADMIN_PASSWORD \
  --set-string env[1].value='ChangeMe123!'

After install, port-forward to access the gateway:

terminal

kubectl port-forward -n deltallm svc/deltallm 4000:4000
curl http://localhost:4000/health/liveliness
# Open the admin UI at http://localhost:4000

The eval profile is for testing only. It uses bundled databases with default passwords. Never use it for production traffic.

Production Setup

For real workloads, use external PostgreSQL and Redis, and store secrets in Kubernetes secrets — not in Helm values.

1. Create secrets

terminal

# App secrets (master key + salt)
kubectl create secret generic deltallm-app-secrets \
  --namespace deltallm \
  --from-literal=master-key="$DELTALLM_MASTER_KEY" \
  --from-literal=salt-key="$DELTALLM_SALT_KEY"

# Runtime secrets (database, Redis, provider keys)
kubectl create secret generic deltallm-runtime-secrets \
  --namespace deltallm \
  --from-literal=DATABASE_URL='postgresql://user:pass@postgres:5432/deltallm' \
  --from-literal=REDIS_URL='redis://redis:6379/0' \
  --from-literal=OPENAI_API_KEY='sk-...'

2. Configure values

terminal

# values-custom.yaml
postgresql:
  enabled: false   # using external database

redis:
  enabled: false   # using external Redis

secret:
  existingSecret: deltallm-app-secrets

runtime:
  database:
    existingSecret:
      name: deltallm-runtime-secrets
      urlKey: DATABASE_URL
  redis:
    existingSecret:
      name: deltallm-runtime-secrets
      urlKey: REDIS_URL

envFrom:
  - secretRef:
      name: deltallm-runtime-secrets

3. Install

terminal

helm install deltallm deltallm/deltallm \
  --namespace deltallm \
  --create-namespace \
  -f values-custom.yaml

High Availability

The production profile gives you enterprise-grade resilience out of the box:

3 replicas — survives node failures
Horizontal Pod Autoscaler — scales based on CPU/memory
Pod Disruption Budget — prevents all replicas from going down during maintenance
Topology spread — distributes pods across availability zones
Soft anti-affinity — avoids putting all replicas on the same node

terminal

helm upgrade --install deltallm deltallm/deltallm \
  --namespace deltallm \
  -f helm/values-production.yaml \
  -f values-custom.yaml

Ingress and TLS

Expose the gateway externally with an ingress controller:

terminal

# Add to your values-custom.yaml
ingress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: llm-gateway.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: llm-gateway-tls
      hosts:
        - llm-gateway.example.com

Set long proxy timeouts (600s+) because LLM requests can take several seconds, especially for long completions or tool calls.

Monitoring with Prometheus

Enable the ServiceMonitor for automatic Prometheus scraping:

terminal

# Add to your values
prometheus:
  serviceMonitor:
    enabled: true

DeltaLLM exposes 20+ Prometheus metrics including request counts, token usage, latency histograms, cache hit rates, and deployment health — all at /metrics.

Seeding Models from Config

You can bootstrap model deployments in the Helm values instead of configuring them through the admin UI:

terminal

config:
  model_list:
    - model_name: gpt-4o
      deltallm_params:
        model: openai/gpt-4o
        api_key: os.environ/OPENAI_API_KEY
        timeout: 300
      model_info:
        mode: chat
  general_settings:
    model_deployment_source: hybrid

Troubleshooting

Symptom	Likely cause	Fix
Pod exits during startup	Missing master-key or salt-key	Check `secret.existingSecret`
Can't connect to PostgreSQL	Wrong DB secret or bundled PG disabled	Check `runtime.database`
Can't connect to Redis	Wrong Redis URL or missing auth	Check `runtime.redis`
Provider calls fail	Missing provider env vars	Add them via `envFrom`
Config change didn't roll pods	External secret changed outside Helm	Restart the deployment manually

Deploying an LLM Gateway on Kubernetes with Helm

Three Deployment Profiles

Quick Start: Evaluation Install

Production Setup

1. Create secrets

2. Configure values

3. Install

High Availability

Ingress and TLS

Monitoring with Prometheus

Seeding Models from Config

Troubleshooting

Try DeltaLLM

Related Articles

LLM Load Balancing with DeltaLLM Model Groups

Share AI Models and Prompts With Your Team—Without Sharing Provider Keys

LLM Observability: Prometheus Metrics and Monitoring for AI Gateways