14 Best Token optimization techniques for production AI workloads

The fastest token optimization win in production is reordering your prompt so the static part comes first, then turning on prefix caching. Cache reads bill at roughly a tenth of the input price on most frontier models; the change takes 4 hours max, and it costs you nothing in output quality. Everything after that involves a tradeoff, and the size of each tradeoff is what this article puts numbers on.

Here is the number that should set your expectations: One senior AI engineer took a production RAG chatbot from $3,700 a day in token spend, about $1.4 million a year, down by 90%, and the single biggest contributor was not a clever prompt. It was replacing an LLM-as-a-judge step with a reranker model and a pair of score thresholds.

What you get in this blog that you won’t find anywhere else: a measured saving and a measured quality cost for each of the 14 techniques, the exact provider thresholds that make caching fail, and the failure modes that only show up once the technique is running in production.

What are you paying for on every single request?

Three mechanics explain nearly all production token spend. If you understand these, you can evaluate any optimization tip you read without testing it first.

Models are stateless, so conversations are re-sent in full

A model stores nothing between calls. "Having a conversation" means resending the entire history, in order, every turn. Todd Toller, a senior cloud solution architect at Microsoft, put the compounding plainly in a GitHub Copilot token optimization session: loop one sends the system prompt, the tool definitions, and your message; loop two sends all of that plus the previous response plus the new input; and it keeps growing.

For agents, this is the whole ballgame. Anthropic's own measurement: "agents typically use about 4x more tokens than chat interactions, and multi-agent systems use about 15x more tokens than chats". The sentence just before it is the one worth pinning to a wall: "There is a downside: in practice, these architectures burn through tokens fast."

Output tokens cost several times what input tokens cost

GitHub's own cost model weights them explicitly. Their Effective Tokens formula is ET = m × (1.0 × I + 0.1 × C + 4.0 × O), where I is input, C is cache reads, and O is output. Output carries 4x the weight of input, and 40x the weight of a cache read. A technique that trims 200 output tokens beats one that trims 700 input tokens.

More context does not mean better answers past a point

This is the part that turns token optimization from a finance exercise into an engineering one. Chroma’s technical report tested 18 models across 8 input lengths and 11 needle positions, 194,480 LLM calls in total, and found that models "do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows".

The numbers elsewhere are harder still. The NoLiMa benchmark evaluated 13 models that all claim 128K-token support: at 32K, 11 of the 13 dropped below 50% of their short-context baseline, and GPT-4o fell from 99.3% to 69.7%.

NVIDIA's RULER benchmark found that of 17 long-context models, "only half of them can maintain satisfactory performance at the length of 32K".

And position inside the window changes the answer. The TACL paper Lost in the Middle found that performance "is often highest when relevant information occurs at the beginning or end of the input context" and falls away when the answer sits in the middle. Keep the size of the fall: GPT-3.5-Turbo dropped by more than 20 percentage points on multi-document QA, scoring below its own closed-book baseline of 56.1%. Those documents made the model worse than having no documents at all.

Put those together, and you get the governing idea for everything below. Trimming context is not purely a cost move. Past a certain fill level, it is a quality move that happens to be cheaper, which is why Toller frames the goal as "instead of counting tokens, we want to make every token count."

1: What can you change in billing before touching any code?

These two techniques cost you nothing in quality. Do them first, in this order.

1. Prefix caching, and the three thresholds that make it fail

Providers match the longest identical prefix of your prompt against a cache and bill those tokens at a steep discount. What you get: cache reads at 0.1x the base input price on most Anthropic models, with 0.05x and 0.025x on some; OpenAI advertises cached input "discounted up to 95%"; Gemini's cached rate on the pricing page is exactly 10% of the input rate; DeepSeek's cache hit runs at about 2% to 3% of the miss price.

Matching stops at the first byte that differs. One variable token at the top of your system prompt, a timestamp or a user ID, costs you the entire cache behind it.

Three thresholds decide whether any of this reaches your bill:

Three traps worth naming, because each one costs real money:

Anthropic publishes a different minimum per model, and the order is not by release date. Opus 4.5 and 4.6 need 4,096 tokens; the newer 5-series models need 512.

2. Batch and off-peak windows for anything that is not interactive

All three major providers have converged on the same offer: 50% off, 24-hour turnaround. OpenAI's Batch API gives "50% lower costs" with "a clear 24-hour turnaround time". Anthropic’s Message Batches API applies "a 50% discount on both input and output tokens", with most batches completing within an hour. Gemini's Batch API processes "at 50% of the standard cost" against a 24-hour SLO.

The discounts stack with caching, with one caveat Anthropic states directly: "because batch requests are processed asynchronously and concurrently, cache hits are provided on a best-effort basis."

DeepSeek adds a second axis that none of the 15 pages we read mentions. Its off-peak rates are half the peak rates, with peak hours defined as 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays. For a nightly embedding or evaluation job, the scheduler is a pricing decision.

Who should skip this: anything a human is waiting on. Batch is for data labelling, bulk summarisation, embedding generation, offline evals and overnight analytics.

2: Stop sending every request to the same model

Price differences between model tiers are larger than almost any optimization you can perform within a single model. Toller cites roughly a 24x gap between a flagship reasoning model and a small one. FrugalGPT's authors found vendor prices "differ by two orders of magnitude" across the market.

3. Route before the call

A lightweight classifier or rule set inspects the request and sends it to the cheapest model that can handle it.

What it saves, measured: the LMSYS RouteLLM team trained four routers on Chatbot Arena data and reported: "cost reductions of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K as compared to using only GPT-4, while still achieving 95% of GPT-4’s performance". Note the spread: 85% on open-ended chat, 35% on grade-school maths. Your savings depend entirely on how many of your requests are genuinely easy.

The tradeoff: routing commits before inference, so a misrouted hard task comes back wrong, and you pay twice. Routing works where the request shape predicts difficulty well. It fails where it does not.

4. Cascade with a validation gate

Try the cheap model first, validate the output, escalate only on failure. This is the pattern FrugalGPT formalised, and the results are the strongest in the literature: "FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost".

The gate is the whole technique. A cascade without a reliable validator is just two bills. Use something deterministic wherever you can: JSON schema validation, a compile step, a unit test, a regex on the expected shape. Confidence scores from the cheap model are the weakest gate because that is exactly what a small model is bad at.

When a cascade loses money: when most of your traffic is hard. You pay the cheap attempt plus the expensive retry on every difficult request. Measure your easy-task fraction before you build this.

5. Set a reasoning-effort budget per call

Reasoning tokens are billed as output, which means they carry the 4x weight in GitHub's model. Every major agent tool now exposes an effort dial, and the default is where it usually stays for every call in a session.

The practical rule: planning, architecture, and hard debugging earn high effort. Renaming a variable, writing a commit message, formatting JSON, and summarising a diff do not. ML.ai exposes five levels, low through max with medium as the default, and lets you change it per message rather than per session, so a single conversation can spend heavily on the design step and cheaply on the twenty mechanical steps that follow.

One more lever worth checking in your own tooling: GitHub's task-aware automatic model selection carries a 10% token cost discount on top of picking a cheaper model, according to Toller's session.

3: Why is context engineering the biggest lever in agent workloads?

If you run agents, this strategy is where the money is. Anthropic’s analysis of their own research agent found that "token usage by itself explains 80% of the variance" in performance on the BrowseComp evaluation. Context is not just the cost. It is also the quality.

6. One task per context window

Start a fresh context for each distinct phase of work rather than carrying research context into implementation.

Why it works: recency bias. Toller's measurement is that past roughly 50% of window capacity, the model starts favouring the end of the conversation over the system instructions and the original prompt, and "you don’t want to steer a bloated session with around 80% of your context filled." Running research, planning and implementation in one window carries every irrelevant file read through every later turn, and you pay for all of it on every turn.

The cost of the habit, measured: across enterprise prompts, Toller reports that 60% to 70% of always-loaded tokens carry no information that changes the model’s answer, and that a single cleanup pass produced about a 56% reduction in input tokens and a 41% faster time to first token, with roughly 30% fewer re-asks.

Tokens do not accumulate across sessions, so a discarded context costs nothing. Clear aggressively between phases.

7. Prune tool results before they reach the context

Filter API and tool responses down to the fields the model needs, in code, before the response enters the conversation.

A worked example from production: the PropTech Founder case had an LLM-as-a-judge step receiving 30 to 50 property listings as full JSON objects. Looking at what the call needed to decide relevance, "there were maybe one or two fields too many there", and trimming those fields alone meaningfully cut the most expensive call in the system. At high traffic, a field is not a field. It is a field times every object times every request.

The format decision is worth real tokens too. The open TOON benchmark measures the same datasets across formats with the o200k_base tokenizer. On a mixed corpus: pretty-printed JSON 4,308 tokens, YAML 3,487, compact JSON 2,892. Minifying JSON alone cuts about a third. On purely tabular data the gap is larger: for 100 uniform employee records, CSV came in at 47,153 tokens against pretty JSON's 127,061.

One caution the repo itself states, and that you should carry: the accuracy differences between formats in that benchmark overlap within their confidence intervals. The token savings are real. Claims that a format makes the model smarter are not supported.

8. Call tools as code instead of loading every schema

Instead of injecting every tool's full JSON schema into the context on every turn, expose the tools as code the model can read selectively and call by writing a short program.

What it saves, measured: this is the largest single figure in this article. Anthropic’s engineering team reported that moving an agent from direct tool calls to code execution over MCP "reduces the token usage from 150,000 tokens to 2,000 tokens, a time and cost saving of 98.7%". The mechanism is selective loading: the agent lists a directory of available servers, reads only the tool files it needs for the current task, and leaves the rest on disk.

The same problem at a smaller scale: MCP servers return tool descriptions straight into the context window the moment they are activated. Toller singles out the Playwright MCP as the clearest case, powerful for front-end work but costly because screenshots and page reads consume many tokens, and wasteful if it stays always-on for a CSS colour change. Deactivate what you are not using, or scope servers to the specific agents that need them.

ML.ai ships this pattern as a setting. Its code mode has the model call tools by writing a short program rather than receiving every tool schema each turn, and the documented saving is about 1,700 tokens per turn. On a 40-turn agent session, that is 68,000 tokens you never pay for.

9. Push exploration into subagents

Hand a research or search task to a subagent with its own context window, and return only a summary to the main conversation.

The shape of the saving, from Anthropic's own description: "Each subagent might explore extensively, using tens of thousands of tokens or more, but returns only a condensed, distilled summary of its work". Detailed search context stays isolated inside the subagent, and your main thread stays clean.

The tradeoff is real and large. Multi-agent systems use about 15x the tokens of a chat interaction. Anthropic’s own conclusion: "For economic viability, multi-agent systems require tasks where the value of the task is high enough to pay for the increased performance." Subagents are a conditional optimization, not a default. Use them where exploration would otherwise flood the main window, and not for tasks a single turn handles.

ML.ai splits its subagents by capability rather than leaving it to the prompt: Explore is read-only and changes nothing, General can change the project but cannot reach outside it, and Architect and Plan investigate and return an approach without touching files. Delegated runs execute in their own session, so a long exploration never lands in the main conversation's context.

10. Compaction, and the failure mode nobody publishes

This is the technique the category gets wrong, and the evidence against the naive version is the most interesting material we found.

The standard version: at a fixed fill level, say 75% of the window, summarise the conversation so far, keep the last few turns, and replace everything earlier with the summary.

What breaks: the summary is written before anyone knows which detail will turn out to be load-bearing. An independent test run on the Prompt Engineering channel measured it on a warehouse-log task where the agent had to track truck seal codes across 16 days of shift logs. The default summariser "listed 12 seal codes but the problem was that 11 of those didn’t exist anywhere." By the third summary, 14 of 18 codes were fabricated.

The dangerous part is not the error rate. It is the presentation. The summary "looks perfectly confident with the check boxes and the next steps. So the agent has no way to know it’s wrong."

The hidden cost even when it works. On another run the agent got the right answer, but "the reason was not that it remembered things." Instead it "went back and reread every file from disk with grep even though the prompt told it not to do so explicitly." A passing evaluation can conceal a cost regression, because recovery work does not show up as a wrong answer.

What to do instead, in order of effort:

Self-editing context is also not automatically cheaper. In one of the runs above, the self-editing setup "processed about twice as many tokens" as the summarising baseline, and on another "only 10% of the prompt came from the cache."

4: Stop paying twice for the same answer

11. Semantic caching, with an honest hit rate

Embed the incoming query, compare it against stored query vectors, and return a cached response when similarity clears a threshold. Unlike exact-match caching, "how do I reset my password" and "I forgot my password, how do I reset it" hit the same entry.

What the peer-reviewed numbers say. The GPTCache paper ran two experiments. On a favourable all-paraphrase query set: 876 hits out of 1,000 queries, an 87.6% hit rate, with 39 of those hits wrong, a 4.5% false-hit rate. On a realistic mix of 50% related and 50% unrelated queries: 570 hits out of 1,160, a 49.1% hit rate. Hit latency was 0.17 to 0.20 seconds against roughly 3 seconds for a live call.

The caveat the authors publish themselves, which no vendor page repeats: "even the best cache hit rates do not exceed 90% with current embedding models. This means that negative cache hits are noticeable during use. While this may not greatly impact individual users, it would be unacceptable in real production scenarios."

Three things to get right before you ship one:

Where it genuinely shines: high-repetition, low-stakes, read-heavy traffic. In the PropTech Founder case, search queries were "very normally distributed", with a handful of keywords dominating. That distribution is what makes a semantic cache pay for itself.

12. Replace the model with code where the task is deterministic

The cheapest token is the one you never send.

The headline example. In the PropTech Founder system, the most expensive call by a wide margin was an LLM-as-a-judge step reranking 50 property listings per query. Downgrading the model hurt results too much, so the fix was structural:

The result was far fewer JSON objects reaching the expensive call, a much faster application, and in his assessment, "probably one of the biggest cost-saving techniques" in the whole project.

The same logic applies to smaller things. That system used an LLM to expand "ocean view" into related amenities like waterfront and beachfront. A lookup dictionary and a regular expression resolve that instantly and for free. The habit worth building: before writing a prompt, ask whether the transformation is deterministic. If it is, it belongs in code.

One more from the same project: embedding dimensions were cut from 1,024 to 768 with no measurable loss in relevance, reducing both storage and embedding call cost. OpenAI's own guidance is that much beyond 768 dimensions is usually overkill.

5: The output side, where tokens cost 4x

13. Cap and shape the output

Output carries 4x the weight of input in GitHub's cost model, and reasoning tokens bill as output. Three moves, in descending order of effect:

That last one has a measurable ceiling. Your instructions file is itself always-loaded context, so a 2,000-token style guide that saves 50 output tokens per call is a bad trade until your call volume is high.

14. Prompt compression, and why it is last

A small model scores each token in a long prompt for how much it contributes to meaning, then drops the low scorers while preserving the original word order.

The published results: Microsoft's LLMLingua reports "up to 20x compression with little performance loss". LongLLMLingua reports "a 94.0% cost reduction in the LooGLE benchmark" and end-to-end latency acceleration of 1.4x to 2.6x at 2x to 6x compression. LLMLingua-2 reports 1.6x to 2.9x end-to-end latency improvement at 2x to 5x compression.

Why it sits at the bottom of this list anyway. It adds a model to your critical path, the published results come from benchmark corpora rather than your prompts, and if you still have an unpruned instructions file or an uncached prefix, you are compressing waste rather than removing it. Compression is what you reach for when the cheap structural work is done, and your prompts are genuinely long and genuinely necessary.

Which of these should you do, and in what order?

The 14 techniques do not sit on one axis. Some are billing changes with no quality cost. Some buy large savings by taking a measurable risk with your output.

Three patterns come out of assembling all of this.

The reliable savings are structural, not verbal. Every technique in the bottom-right quadrant changes where work happens: which model, which window, which process, which format. Every technique in the top half changes what information survives, and that is where the measured failures are. Rewriting prompts to be shorter is the most commonly recommended tactic in this category and one of the least reliable, because the thing you deleted was sometimes load-bearing.

Quality improvements and token savings point the same direction more often than they conflict: Toller's compounding-error arithmetic makes this concrete: at 99% accuracy per step, a 50-step agent workflow finishes at 61% overall accuracy, and at 95% per step it finishes at 8%. Every failed agent run costs the full token spend of the run, plus the retry, plus the human time.

Deterministic checks pull it back. In Toller’s words: "If after 10 steps you’ve landed at 50% accuracy, the test will fail and bring the agent back on track to 99%." A test suite is a token optimization.

Your maturity decides whether any of this is worth doing: Toller's framing is the most useful filter we found: if you spend $20 a month, saving 50% through every optimization here gets you to $10, which is not worth an engineer's week. If you orchestrate hundreds of asynchronous agents, every percentage point compounds across every run. Measure your spend before you optimize it.

How do you know any of it worked?

The common failure is changing three things at once and attributing the savings to whichever one took the most effort. GitHub’s method is the one to copy, because it is the only published example in this category with a stated procedure:

Their wins, with run counts attached: 62% across 109 runs, 59%, 43%, 37% across 8 runs, 19% across 12 runs. The spread between 19% and 62% on the same platform with the same techniques is the real lesson. There is no universal number, which is why every "cut your costs by 70%" headline should be read as a ceiling someone else hit on a workload that was not yours.

Add one thing GitHub could not: a quality metric. Run a fixed evaluation set before and after every optimization. A token saving you cannot pair with a stable accuracy number is not a saving, it is a deferred incident.

Where does ML.ai fit into this?

Everything above is work. Reordering prompts for cache alignment, routing per step, scoping MCP servers, isolating exploration into subagents, watching the fill level of a window: it is a standing engineering cost on top of the actual product you are building.

ML.ai moves that work into the inference layer. It reads your repository and picks a model for each step rather than sending every call to one tier, which is techniques 3 and 4 applied automatically per step rather than configured per route. Code mode handles technique 8, calling tools by writing a short program instead of receiving every tool schema each turn, for a documented saving of about 1,700 tokens per turn.

The subagent split covers technique 9: Explore runs read-only in its own window, delegated runs execute in their own sessions, and the main conversation never inherits the exploration. Reasoning effort is a five-level dial you set per message, so technique 5 costs you one keystroke.

Two details are worth calling out because they address the measurement gap this article opened with.

A context meter, with thresholds where the research says they belong. It reports tokens used against the model's limit and turns amber at 65% and red at 85%. Given the recency-bias finding that steering degrades past roughly half the window, a visible fill level is the difference between clearing context deliberately and discovering the problem in a bad diff.

Truncation is marked, never silent. Attachments are bounded to 50 KB, 2,000 lines and 2,000 characters per line before they reach the model, and anything trimmed is labelled as trimmed. The silent-failure problem that runs through this whole category, the cache that does not engage and returns no error, the summary that invents 14 of 18 identifiers while looking confident, is the same problem in a different place. Knowing what the model did not see is worth as much as knowing what it did.

The commercial version of the same idea is the pricing: one flat price, and no token counter to watch. For a team where developers currently self-ration their use of an expensive model, the saving is not only in the tokens. It is in the engineering hours that go back into the product.

If you want to see the numbers against your own repository, the ML.ai pricing page has the current plans and the ML.ai documentation covers the context meter, subagent scopes, and code mode in detail.

Conclusion

You now have a measured saving and a measured risk for each of the 14 techniques, the exact thresholds that make caching fail silently, and a method for proving a change worked rather than assuming it did.

Start with the cheapest check in this article: pull one request from your highest-volume route and count the tokens before the first variable byte in your prompt. If that number is below your provider's minimum cacheable length, or if a timestamp sits above your system instructions, you have been paying full price for content that has not changed in months. The fix is an afternoon of reordering, and it costs you nothing in quality.

Then instrument, and only then optimize. If you would rather have the routing, the subagent isolation, and the tool-schema offloading handled in the inference layer instead of becoming four tickets on your board, that is the problem ML.ai is built to take off your plate.

Frequently Asked Questions

What is token optimization?

Token optimization is the practice of reducing the number of tokens a language model processes per unit of useful work, through caching, routing, context management, and replacing model calls with code, without losing output quality. In production, it covers both the billing side (cache reads, batch discounts, model tiers) and the engineering side (what enters the context window and when).

How much can token optimization realistically save?

Published results range from 19% to 98.7% depending on the workload and the technique. GitHub measured 19% to 62% across its own agentic workflows using ordinary context hygiene. Anthropic measured 98.7% on one specific change, moving from direct tool calls to code execution. Treat any single headline percentage as a ceiling someone hit on a workload that is not yours, and measure your own baseline first.

Why is my prompt caching not reducing my bill?

The three most common causes, in order. Your prompt is below the provider’s minimum cacheable length, which is 512 to 4,096 tokens on Anthropic depending on the model and 1,024 tokens on recent OpenAI models, and no error is returned when this happens. Or something variable sits near the top of your prompt, a timestamp or session ID, which stops prefix matching before it reaches your static content. Or your cache entries expire between requests, since default TTLs run from about 5 minutes of inactivity.

Does reducing tokens hurt output quality?

It depends entirely on which technique. Prefix caching, batching and format changes carry no quality cost because the model sees identical content. Routing, cascading and subagent isolation carry a manageable risk if you gate them. Compaction and aggressive prompt compression carry a measured risk: in one documented test a default summariser fabricated 14 of 18 identifiers by the third compaction while presenting the result confidently. Pair every technique in the second and third groups with a fixed evaluation set.

Is a bigger context window a substitute for context management?

No, and the benchmarks are consistent on this. Of 13 models claiming 128K-token support, 11 dropped below 50% of their short-context baseline at 32K tokens in the NoLiMa evaluation. NVIDIA's RULER found only half of 17 long-context models held satisfactory performance at 32K. A larger window raises the ceiling on what you can send; it does not make the model read all of it reliably.

Should I use a semantic cache or an exact-match prompt cache?

Use both, for different things. Prefix caching is provider-side, lossless, and applies to the static portion of every request. Semantic caching is application-side, returns a stored answer for a similar-but-not-identical query, and carries a false-hit rate that the GPTCache authors measured at around 4.5% of hits, with the best hit rates not exceeding 90%. Prefix caching first, always. Semantic caching only where repeated questions are common, and a slightly wrong answer is survivable.

How do I measure token optimization properly?

Log per-call usage with input, output, cache-read, and cache-write tokens separately, then normalise to a single weighted figure so comparisons hold across model tiers. GitHub's published formula weights output at 4x input and cache reads at 0.1x. Set a minimum run count before you compare periods, run a fixed evaluation set alongside, and publish your negative results to yourself so you can spot a workload shift masquerading as a regression.