Engineering guide
Reduce OpenAI API costs without guessing at quality.
Cost work fails when teams optimize an aggregate invoice without knowing which product outcome created it. Start with request-level evidence, then apply the least disruptive control to each expensive pattern.
Summary
To reduce OpenAI API costs, measure actual token usage by feature, remove repeated context, cache deterministic work, route simple tasks to smaller models, cap output tokens, and enforce budgets before requests reach the provider. Validate every change against product-quality tests.
First move
Attribute before optimizing
Largest waste
Repeated unnecessary work
Safety check
Quality regression suite
1. Build a baseline tied to product outcomes.
Record input tokens, output tokens, model, latency, cache state, customer, and feature for every request. Then calculate cost per successful workflow: resolved ticket, generated document, reviewed claim, completed agent task, or another outcome your product can verify.
An average cost per request can be misleading when request sizes vary widely. Compare medians and upper percentiles by feature, and inspect the expensive tail separately.
2. Remove context the model does not need.
Long system prompts, duplicated instructions, full conversation replay, and unfiltered retrieval results are common sources of input cost. Measure each component before deleting it so the change remains explainable.
Summarize older conversation turns, retrieve fewer but better-ranked documents, and keep stable instructions separate from request-specific content where provider prompt caching can reuse them.
3. Cache work that is safe to reuse.
Response caching works best for deterministic requests whose answer is not user-specific or time-sensitive. Include model, prompt content, parameters, and relevant policy context in the cache key. Exclude identifiers only when they do not change the answer.
Track hit rate and avoided provider cost, but also define invalidation. A high hit rate is not useful if stale responses violate product expectations.
4. Route by task difficulty instead of using one default model.
Classify workflows by quality requirement and failure cost. Extraction, classification, rewriting, and simple support intents often have different model requirements from complex planning or high-stakes generation.
Evaluate candidate models against a fixed dataset. Route only after the smaller model meets the acceptance threshold, and retain a fallback for provider or quality failures.
5. Put explicit boundaries on generation.
Set output token limits that match the interface. Ask for structured fields when the product needs fields, not an essay. Stop agent loops with maximum steps, elapsed-time limits, and tool-call budgets.
For background jobs, enforce concurrency and per-key budgets so a retry storm cannot consume the project allowance reserved for user-facing traffic.
6. Enforce a budget before the next provider call.
Alerts create awareness; hard limits contain exposure. Start with an 80% warning, test the application fallback, then enable the 100% block. Keep project, API-key, customer, and feature limits separate where ownership differs.
7. Treat quality as a regression constraint.
Every cost change should run against representative inputs and measurable acceptance criteria. Track task success, structured-output validity, human review rates, retries, latency, and provider cost together.
The cheapest request is not the goal. The goal is the lowest reliable cost for the product outcome, with a controlled failure mode when the budget or provider is unavailable.
Try Halvr
Route a real request through Halvr.
Keep your current SDK. Change the base URL, attach customer and feature metadata, and set a spend limit.