Start with a small job before submitting a whole knowledge base. This walkthrough embeds two sample documents, checks item-level results, and shows how to assess cost without assuming a provider discount.
Gateway batch jobs and provider batch pricing are different
| Mechanism | What it changes | What to verify |
|---|---|---|
| Gateway async queue | When work runs; progress, retry, and result handling | Worker capacity, completion time, failures, and storage cost |
| Embedding micro-batching | Combines eligible inputs into fewer upstream requests | Provider support and measured throughput; token rates may stay unchanged |
| Lower-cost model | Provider charges for the selected model | Output quality and, for embeddings, dimensions and retrieval quality |
| Provider-native async batch | May use the provider's separate batch pricing | Whether the gateway actually submits to that provider's native batch service |
The current DeltaLLM batch documentation describes an internal worker queue. It does not use provider-native async chat batch APIs in this release. Do not assume a provider's advertised batch discount applies to jobs submitted here.
See the batch API and production setup reference for your installed version. Older releases may support embeddings only; chat batches require a version with non-streaming chat support.
Run a two-document embedding batch
1. Prepare the gateway
Complete the Docker setup, then configure an embedding deployment named text-embedding-3-small. The starter chat deployment alone is not enough. Create an application key with access to the embedding target and enough budget for the job.
Enable batching in your existing config.yaml and restart the gateway. Ensure the batch worker is enabled for your deployment. This single-instance evaluation uses local storage; use the production reference when separating API and worker instances.
general_settings:
embeddings_batch_enabled: trueSet DELTALLM_API_KEY to the scoped key. The single-instance Compose address is http://localhost:4002. The curl steps below use jq to read JSON response IDs.
2. Create and upload the JSONL input
Save these two lines as input.jsonl, or download the sample input. Keep the custom_id unique so each result can be matched back to its source document.
{"custom_id":"support-refunds","method":"POST","url":"/v1/embeddings","body":{"model":"text-embedding-3-small","input":"Refund requests are reviewed within two working days."}}
{"custom_id":"support-delivery","method":"POST","url":"/v1/embeddings","body":{"model":"text-embedding-3-small","input":"Customers can track a delivery using the order reference."}}export DELTALLM_BASE_URL=http://localhost:4002
FILE_ID=$(curl --fail-with-body -sS "$DELTALLM_BASE_URL/v1/files" \
-H "Authorization: Bearer $DELTALLM_API_KEY" \
-F "purpose=batch" -F "[email protected]" | jq -er '.id')
printf 'Input file: %s\n' "$FILE_ID"3. Submit the job
BATCH_ID=$(curl --fail-with-body -sS "$DELTALLM_BASE_URL/v1/batches" \
-H "Authorization: Bearer $DELTALLM_API_KEY" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg id "$FILE_ID" '{
input_file_id: $id,
endpoint: "/v1/embeddings",
completion_window: "24h"
}')" | jq -er '.id')
printf 'Batch: %s\n' "$BATCH_ID"Use the supported 24h completion-window value. It is an API setting, not a promise that every job succeeds within that time.
4. Inspect progress and download both result files
curl --fail-with-body -sS "$DELTALLM_BASE_URL/v1/batches/$BATCH_ID" \
-H "Authorization: Bearer $DELTALLM_API_KEY" | jq .Poll with a delay while the job is active. Check status, request_counts, output_file_id, and error_file_id. Download the output file when available and inspect the error file too: a completed job can contain failed items. Match records by custom_id rather than relying on line order.
For a complete client that polls with a time limit and downloads both files, use the Python embedding batch example. It uses requests; install it with python -m pip install requests, then run python embedding-batch.py input.jsonl.
Estimate savings with explicit assumptions
Suppose a workload uses 10 million input tokens. An existing embedding model costs an illustrative $0.10 per million tokens and a suitable alternative costs $0.04. These are arithmetic assumptions, not current provider prices or measured DeltaLLM savings.
Existing model: 10 million tokens × $0.10 / million = $1.00
Alternative: 10 million tokens × $0.04 / million = $0.40
Difference: $1.00 − $0.40 = $0.60, or 60%
Same model and same token rate in an async queue:
10 million tokens × $0.10 / million = $1.00 before extra costsThe 60% comes from the assumed model price difference, not the queue. Include retries, gateway compute, storage, and any output-token charges for chat workloads. Changing an embedding model may require re-indexing and retrieval-quality testing.
For an actual comparison, process the same document set with the same chunking and model settings. Record billed tokens, failed items, retry count, elapsed time, and worker cost. Report cost per successfully processed document alongside quality, rather than comparing only nominal token prices.
The broader cost-management guide covers budgets and routing. A budget controls spend; it does not itself make each request cheaper.
Before scaling beyond the example
- Protect interactive capacity: bound worker concurrency and tune model-capacity allocation before sharing a provider with live chat traffic.
- Keep policies consistent: batch execution rechecks access and budgets. Permanent policy failures are not fixed by retrying.
- Use shared storage: separate workers and API replicas need shared artifact storage, such as S3-compatible storage, and distributed coordination.
- Handle partial success: inspect failed items and resubmit only the intended failures. Avoid replaying the entire successful job.
For chat batches, change the endpoint and each JSONL request body to /v1/chat/completions, use a permitted chat target, and omit streaming. Current documentation excludes MCP tools in chat batch requests. Do not mix embedding and chat endpoints in one job.