If you're calling an LLM with the same prompt twice, you're paying twice for the same answer. Classification tasks, FAQ bots, and template-based prompts often send identical requests — and every one costs you tokens, time, and money. A cache at the gateway level intercepts these duplicates before they reach the provider.
How Gateway Caching Works
DeltaLLM sits between your app and the LLM provider. When caching is enabled, it stores responses keyed by the request contents. The second time the same request comes in, the gateway returns the stored response instantly — no provider call, no tokens consumed, no latency.
The cache works on these endpoints:
/v1/chat/completions— including streaming replay/v1/completions/v1/responses/v1/embeddings— especially impactful for repeated embedding lookups
Choose a Cache Backend
| Backend | Best for | Notes |
|---|---|---|
memory | Local dev or single instance | No external dependency. Data is lost on restart. |
redis | Production with multiple instances | Shared cache across replicas. Best default. |
s3 | Long-lived object-backed cache | Use when cache entries need to persist beyond Redis TTL. |
Start with memory for testing, then switch to Redis for production:
# config.yaml — start simple
general_settings:
cache_enabled: true
cache_backend: memory
cache_ttl: 3600
cache_max_size: 10000# config.yaml — production
general_settings:
cache_enabled: true
cache_backend: redis
redis_url: os.environ/REDIS_URLHow Cache Keys Work
DeltaLLM builds cache keys from:
- The full request payload (messages, model, parameters)
- The target model name
- The authenticated request scope (API key)
This means two different API keys sending the same prompt do not share cache entries by default. This is intentional — it prevents cache leakage between tenants in multi-team setups.
Verify Cache Hits
Every response includes headers that tell you whether caching was involved:
| Header | Meaning |
|---|---|
x-deltallm-cache-hit: true | This response came from cache |
x-deltallm-cache-key | The cache key used (useful for debugging) |
# Quick test: send the same request twice
curl -s http://localhost:4002/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"What is 2+2?"}]}' \
-D - 2>/dev/null | grep -i cache
# Second request should show: x-deltallm-cache-hit: truePer-Request Controls
Not every request should be cached. DeltaLLM gives you fine-grained control:
| Control | Effect |
|---|---|
metadata.cache: false | Skip cache entirely (no lookup, no write) |
metadata.cache: "no-cache" | Skip lookup but write the fresh response |
metadata.cache: "no-store" | Try lookup but don't store the new response |
metadata.cache_ttl: 120 | Override TTL for this request |
metadata.cache_key: "shared" | Use a custom key (share cache across API keys) |
You can also use standard HTTP headers:
# Skip cache with HTTP headers
curl http://localhost:4002/v1/chat/completions \
-H "Cache-Control: no-cache" \
-H "Authorization: Bearer $KEY" \
...What Still Gets Enforced
Even on a cache hit, DeltaLLM still runs:
- Authentication — the request must have a valid API key
- Budget checks — cached responses still count against spend tracking
- Rate limits — cache hits count toward request limits
This means caching doesn't bypass your access controls — it only skips the provider call.
When to Cache
| Use case | Cache? | Why |
|---|---|---|
| FAQ classification | Yes | Same questions produce same categories |
| Embedding generation | Yes | Same text = same vector, always |
| Customer support chat | Maybe | Cache common questions, skip unique conversations |
| Creative writing | No | You want unique outputs each time |
| Code generation | No | Context-dependent, rarely identical |