11 LLM caching strategies that actually work in production

The best LLM caching strategy starts with a byte-stable prompt prefix, adds versioned application caches only where reuse is safe, and treats semantic reuse as a prediction that must earn trust before it answers. Teams that get the layout right report prompt-cache hit rates above 85% and input bills cut by more than three quarters.

Two production numbers show the size of the prize. The VS Code team holds a prompt-cache hit rate of about 94% for GitHub Copilot agent sessions on Anthropic models, and Deriv cut its production agent's input-token cost by 77% at an 85.8% hit rate without changing the model or removing context. Both results came from how the prompt was ordered and where cache boundaries were placed.

Caching can also go wrong. It can return stale answers, leak data across tenants, or produce a bill dominated by cache writes. The difference is rarely the cache product. It is the cache boundary, key, lifetime, invalidation policy, and measurement loop.

This blog covers 11 production strategies across four cache layers, each with a team or study that measured it and the steps to copy it. It also shows the hit rate you need before caching pays at all, and what the same cached tokens cost across 61 provider listings.

What are the four cache layers in an LLM system?

Teams often use "LLM cache" to mean several different mechanisms. They solve different problems.

A production system can use all four. Measure each layer separately, because a "cache hit" at one layer has different economics and quality implications at another. Prompt and KV caches still run generation, so the answer is always newly computed. Response and semantic caches return a stored answer, which is why they need stricter keys.

What do production teams actually report from LLM caching?

The strongest results come from teams that treated caching as an engineering discipline with a metric, not a switch they turned on. The six results below are the anchors for the strategies that follow.

Independent measurement puts the ceiling lower than vendor headlines. A PwC research team ran more than 500 agent sessions with 10,000-token system prompts across OpenAI, Anthropic, and Google and measured 41% to 80% lower API cost from prompt caching. It also found that caching the full context, tool results included, made GPT-4o's first token 8.8% slower, because the cache kept writing entries that were never read.

Which LLM caching strategies should you implement?

Instrument first, stabilize the repeated work, and add higher-risk reuse only after the simpler layers are reliable. The order below follows that sequence.

1. Measure cache economics before changing prompts

Start with four numbers for each request class: uncached input tokens, cache-write tokens, cache-read tokens, and time to first token. Add the cache hit rate and the age of the reused entry. Then calculate cost per completed task, not only cost per request. A cheap cached response that fails verification and triggers a retry is expensive.

Current provider APIs expose cache usage differently. OpenAI reports cached tokens in the usage object; Anthropic separates cache_creation_input_tokens and cache_read_input_tokens; Amazon Bedrock returns cacheWriteInputTokens and cacheReadInputTokens. Build an internal schema that normalizes these fields so provider changes do not break your dashboard.

Who did it: Deriv tracks one number per request, cached input tokens divided by total input tokens, and watches it after every prompt or framework change. That feedback loop is how the team caught ordering problems and pushed its hit rate to 85.8%. The VS Code team built a Cache Explorer that draws each prompt as a stacked bar of its parts, so an engineer can see which segment broke the prefix.

Log the cache lifetime, too: Anthropic returns a per-request split between 5-minute and 1-hour cache writes. A developer used exactly that field to analyze 119,866 API calls across two machines and showed that Claude Code's writes moved from the 1-hour bucket to the 5-minute bucket around 6 to 8 March 2026, with no change on the user's side. Without that field in the dashboard, the shift would have looked like ordinary usage growth.

def log_cache(usage, request_class: str):
    read  = usage.cache_read_input_tokens or 0
    write = usage.cache_creation_input_tokens or 0
    fresh = usage.input_tokens or 0       # tokens after the last breakpoint
    total = read + write + fresh
    hit_rate = read / total if total else 0.0
    ttl_mix  = usage.cache_creation       # ephemeral_5m vs ephemeral_1h
    emit("cache.hit_rate", hit_rate, tags={"class": request_class})
    emit("cache.write_share", write / total if total else 0.0, tags={"class": request_class})

Track the distribution as well as the average. A 70% global hit rate can hide one tenant at 98% and another at 5%. Segment by tenant and workflow, then by model, prompt version, toolset, and region.

Implementation checklist

For the broader cost model, see How to reduce LLM API costs and LLM inference cost.

2. Keep the shared prompt prefix byte-stable, ordered by volatility

Provider prompt caches match from the beginning of the request, so the first changed token ends the reusable prefix. Order the prompt from the content that changes least to the content that changes most:

Avoid injecting dates, random IDs, fluctuating tool descriptions, or unordered JSON near the front. Canonicalize serialization, sort stable collections, and pin prompt-template versions.

Who did it: Deriv made four changes to its production assistant: it separated static instructions from request-dependent context, moved shared context ahead of conversation history, sorted active and loadable skills by name before rendering them, and started monitoring hit rate. The reordering mattered most. Shared context that sat after the conversation history could never extend the common prefix, because every conversation diverged before reaching it.

Result: An 85.8% hit rate and 77% lower input-token cost. Deriv puts the gap in plain terms: for every $10,000 of monthly input spend without caching, its well-structured prompt saves about $7,700, while a poorly structured one with caching switched on (around a 20% hit rate) saves about $1,800.

The rule is strict at the byte level. In a Microsoft Foundry demo, cached tokens appeared only once the system prompt passed 1,024 tokens, then grew in 128-token steps (1,024, then 1,152). Rewording the system message and changing its order, while keeping the meaning, dropped cached tokens to zero.

OpenAI's current guide documents a minimum cacheable prefix of 1,024 visible tokens for GPT-5.6 and later. Anthropic describes the same full-prefix behavior across tools, system content, and messages, with minimums from 512 to 4,096 tokens depending on the model. Stable-prefix-first is the most consistent rule across current provider documentation.

3. Place explicit cache breakpoints around expensive stable blocks

Automatic prefix caching is useful, but explicit breakpoints make the intended boundary visible. Place them after blocks that are large, reused, and versioned independently: a system prompt, a tool registry, a repository map, a policy manual, or a long document. Anthropic allows up to four breakpoints per request, and since GPT-5.6 OpenAI also supports explicit breakpoints with up to four cache writes per request.

Choose breakpoints from request traces, not from visual sections in a prompt template. A 5,000-token tool schema reused across every request deserves a boundary. A 200-token example changed on every deploy does not.

Who did it: The VS Code team reworked its Anthropic caching to spend all four breakpoints deliberately: one at the end of the tool definitions, one at the end of the system prompt, and a pair of rolling anchors on the two most recent cacheable messages. The older rolling anchor is a safety net. If the newest anchor misses because a slow tool call let it lapse, the older one still serves a hit for everything before it, so the agent loses one exchange instead of the whole conversation cache.

Result: A steady rise of a few percentage points, to a hit rate of about 94% for agentic workloads.

tools[-1]["cache_control"]  = {"type": "ephemeral"}   # 1: end of tool definitions
system[-1]["cache_control"] = {"type": "ephemeral"}   # 2: end of system prompt
messages[-1]["content"][-1]["cache_control"] = {"type": "ephemeral"}   # 3: newest turn
messages[-3]["content"][-1]["cache_control"] = {"type": "ephemeral"}   # 4: older anchor

Place each breakpoint on the last block that is identical across the requests you want to share a cache. Anthropic's docs note that a lookup walks back at most 20 content blocks, so a turn that adds many blocks needs an earlier anchor. Treat each breakpoint as a versioned interface with an owner, metric, and invalidation rule.

4. Keep agent threads append-only and load tool definitions on demand

Agent loops repeatedly send the system prompt, tools, repository context, and prior messages. Small edits to earlier messages can invalidate the entire downstream prefix. Preserve previous messages exactly and append new turns. When a user changes the goal mid-task, append a message that says so instead of editing the original request.

When context must be compacted, create a durable checkpoint instead of rewriting the thread on every turn. Summarize once, version the summary, and append from there.

The largest remaining source of prefix churn is the tool list. Tool definitions sit near the very front of the prompt, so adding, removing or reordering one resets everything after it. The fix is to stop sending every definition on every turn.

Who did it: The VS Code team moved GitHub Copilot to tool search. The model sees only the name and description of deferred tools, and full schemas load only when the model asks for them. Loaded tools are appended at the end of the context, so the cached prefix is never rewritten. A small core set (read and edit files, run commands, search the workspace) stays loaded.

Result: For the median Copilot user on Anthropic models, prompt and total tokens per session each fell about 18%. On GPT-5.4 and GPT-5.5, session token use fell 8.97% and 10.92%. Anthropic's own test with up to 502 MCP tool definitions found that tool search kept cost flat while loading every definition nearly doubled it, a 45% saving at the largest catalog.

If you cannot use tool search, keep the full tool list fixed and restrict each turn with a setting that sits outside the prefix, such as OpenAI's `allowed_tools`. Our coding agent takes a related approach: an experimental code mode has the model call tools by writing a short program instead of receiving every schema each turn, which saves roughly 1,700 tokens per turn.

5. Choose cache lifetime from observed request gaps

Cache lifetime (TTL) is an economic decision. Measure the time between requests that share a prefix, then compare cache-write premiums with read discounts and storage fees.

Anthropic's default prompt cache lasts five minutes and refreshes on each hit; its one-hour option charges 2x the base input price to write. OpenAI's GPT-5.6 and later default to a 30-minute minimum lifetime. Gemini explicit caches let the caller set a TTL and charge for storage, while implicit caching is automatic.

Who measured it: Anthropic ran a triage agent three ways. Keeping the 5-minute cache warm with keepalive requests cost 13% to 20% less per session than the 1-hour cache when pauses lasted minutes; only when pauses approached 45 minutes did the 1-hour cache win, by about 12 cents a session. That test ran on Claude Fable 5.1, where cache reads cost 0.025x the input price, so keepalives are cheaper there than on models with a 0.1x read price.

Anthropic's recipe is to resend the previous request with max_tokens set to 0 within four minutes of its start.

Who else did it: The VS Code team switched on 24-hour prompt cache retention for supported OpenAI models. For GPT-5.4, the cache hit rate rose 137% at 20 to 30 minute gaps and 919% at 40- to 60-minute gaps (10.2 times the previous rate). Open-source coding agent Aider exposes the same idea as a `--cache-keepalive-pings` flag.

Don't send keepalive traffic by default. It is justified only when the probability and value of another request exceed the keepalive cost. Long TTLs also extend the period in which stale policy, documents, or permissions can be reused.

6. Cache documents and repository snapshots by content hash

Large documents, code indexes, and repository maps are good cache candidates because they are expensive to resend and easy to version. Build the cache identity from the actual content, not a mutable filename or branch name.

A practical document key includes:

tenant_id + source_id + content_hash + parser_version + access_policy_version

For repositories, include the commit SHA or a hash of the selected files, the indexing configuration, and the tool-schema version. When a source changes, create a new immutable entry rather than mutating the old one. Garbage-collect unreachable versions after the longest allowed retention window.

Who did it. Anthropic used this pattern to make Contextual Retrieval affordable: each document is loaded into the prompt cache once, and context for every chunk is generated against the cached copy. With 8,000-token documents and 800-token chunks, the one-time cost came to $1.02 per million document tokens. AWS shows the same economics for Q&A: a 10,000-token contract with 10 questions saves about 75% of the document's input cost when all questions arrive inside the cache lifetime.

This makes invalidation explicit and auditable. It also prevents a user who loses access from receiving an answer generated from a previously authorized document.

7. Cache retrieval and tool outputs with data and permission versions

Many agent calls spend more time and money in retrieval, SQL, search, embeddings and external APIs than in generation. Cache these results separately from the final answer. Embedding calls for documents that have not changed are the easiest win, because the same text always produces the same vector for a given model.

The key should include the normalized query, data snapshot, authorization scope, tool version, locale, and any filters that affect the result. Invalidate on source updates, deletion, permission changes, or a shorter domain-specific freshness deadline.

Do not share retrieval results across users merely because the query text matches. The OWASP RAG Security Cheat Sheet warns that response caching creates cross-user leakage, stale permission enforcement, and persistent poisoning risks. It recommends scoping the cache by user, tenant and permission level, invalidating when a source document changes or its permissions change, and not caching highly sensitive data at all.

Scope the entry before similarity or deduplication logic runs. Walmart's production cache does this by partitioning the cache per tenant so business units share infrastructure without sharing entries.

8. Use exact response caching for deterministic work

Exact response caching is the simplest application cache and should be the first response-level strategy you try. It works best when the task is deterministic, inputs are normalized, and the response can be invalidated precisely.

Strong candidates include classification, entity extraction, policy checks, fixed-format transformation, embeddings, and low-temperature summaries of immutable text. Weak candidates include open-ended writing, advice with changing facts, and answers dependent on user history or live tools.

Build the key from every input that can change the answer. A useful test from one caching walkthrough: the key must be stable (same input, same key), canonical (whitespace and key order do not matter), and specific (model, temperature and anything else that changes the output is included).

tenant + model + model_revision + prompt_version + tool_schema_hash +
  normalized_input + temperature + data_version + policy_version

Store the output, structured validation result, provenance, creation time, and expiry. If you cannot enumerate the factors that affect correctness, the response is not safe to cache. Walmart pairs its semantic layer with an in-memory exact-match tier for exactly this reason: exact lookups are faster and carry less risk.

9. Use verified semantic caching with bypass rules and two thresholds

Semantic caching can reuse an answer when a new query is close in meaning to a prior query. It creates the largest correctness risk because similarity is not equivalence. Nathan Kolano, an applied AI engineer at Redis, frames it well: exact caching is deterministic, semantic caching is probabilistic, so treat the cache like a classifier and track precision, recall and F1 against a labelled test set.

Use two thresholds instead of one. Reuse only above a high-confidence threshold. Send borderline matches to a lightweight judge that checks whether the candidate answer remains valid for the new query, tenant, time window, and source set. Below the lower threshold, call the full model.

Who did it: Walmart published waLLMartCache, its production semantic cache built on GPTCache. It spreads the cache across multiple nodes with a distributed eviction manager, uses Redis as an in-memory tier, partitions entries per tenant, and preloads frequently asked questions at boot. Its decision engine sends code and time-sensitive queries straight to the model instead of the cache.

Result: Redis CEO Rowan Trollope described two deployments in a 2026 interview: a proof of concept for one of the largest US healthcare providers during open enrollment reached a 95% cache hit rate on consumer questions, and a Fortune 50 company using a semantic cache for TV voice search cut its LLM charges by 70%. Both workloads are short, single-turn and highly repetitive, which is where semantic caching works best.

The research backs a verification step. The Krites paper keeps the critical path unchanged and asynchronously asks an LLM judge whether a near-miss static answer is acceptable; approved matches are promoted, which served up to 3.9 times more curated answers. SISO uses centroid-based entries, locality-aware replacement and adaptive thresholds and reports up to 1.71 times higher hit ratios. Treat these as evidence for architecture choices, not production guarantees.

The threshold is a security setting: The NDSS 2026 study Cache Me, Catch You poisoned a semantic cache at GPTCache's default similarity threshold of 0.8. Injected entries reached an average hit rate of 66%, rising to 72% once more than 500 were injected, and generating 500 poisoning queries cost the attacker about $0.75. Adding an LLM check after each hit cut the attack hit rate to 27%.

10. Cache plans and structured intermediate results in agents

Agent answers often depend on live state, but parts of the reasoning pipeline are reusable. Cache a validated plan, schema mapping, repository map, tool-selection result, or test-discovery result rather than the final user-facing answer.

Structured intermediates are easier to validate and invalidate. A plan can be keyed by task class, repository version, policy version, and available tools. The agent can verify preconditions before executing it. If the environment changes, the plan is discarded without risking a stale final answer.

Who measured it: Agentic Plan Caching extracts plan templates from completed runs, matches new requests by keyword, and adapts the template with a small model. Across several real agent applications it cut cost by 50.31% and latency by 27.28% on average while maintaining task performance. Redis showed the same idea at sub-question level in a deep-research demo: after a question was split into sub-questions and one had been answered before, a follow-up needed two LLM calls and about 300 tokens instead of five calls and about 750.

The operational lesson is narrower and more useful: reuse stable decisions only after checking the state assumptions on which they depend.

11. Use prefix-aware KV serving and isolate every tenant

Self-hosted inference gives teams direct control of the KV cache. Before PagedAttention, serving systems reserved one contiguous block per request, so only 20% to 40% of that memory held real tokens. The vLLM paper stores the cache in small blocks, cuts waste to under 4%, lets requests share blocks for a common prefix, and reported two to four times higher throughput at comparable latency.

SGLang's RadixAttention reuses KV cache across calls through a radix tree and reported up to 6.4 times higher throughput. The two results use different systems and tests, so do not rank them against each other.

Larger deployments now split prefill and decode onto separate GPUs and move the KV cache between them over RDMA. vLLM exposes this through a NIXL connector, and schedulers such as llm-d coordinate which prefill and decode workers handle a request. This prefill and decode walkthrough explains the transfer path; the gain disappears if the KV cache moves over ordinary TCP.

In production, the scheduler should group requests with a shared prefix, protect hot entries from premature eviction, and cap per-tenant occupancy. Use cryptographic, salted hashes over canonical serialized prefixes. Never let approximate matching or a truncated hash become the tenant boundary.

The NDSS 2026 study Cache Me, Catch You catalogued six cache-related attack vectors in LLM serving frameworks, including forged prefix-cache collisions and multimodal collisions used to slip content past moderation. The authors disclosed to vLLM, SGLang, GPTCache, AIBrix, rtp-llm and LMDeploy, and vLLM, GPTCache and AIBrix adopted the proposed fixes. Their tools and advisories are public.

Isolation does not have to mean giving up sharing. CacheSolidarity monitors cross-user reuse and isolates only suspicious prefixes, and reports up to 70% higher cache reuse and 30% lower latency than isolating every user.

How do you calculate whether prompt caching saves money?

Use the provider's write multiplier, read multiplier, and storage cost. For a repeated prefix of size T, requested N times:

uncached cost units = N x T
cached cost units   = write_multiplier x T + (N - 1) x read_multiplier x T + storage

OpenAI's current GPT-5.6 example uses a 1.25 times cache-write multiplier and a 0.1 times cache-read multiplier. One write followed by one hit costs 1.35 input-cost units instead of 2. Ten requests cost 2.15 units instead of 10, before any other charges.

Production traffic is not one clean prefix, so the more useful question is what hit rate you need. If every miss writes a new entry, effective input cost per token is hit_rate x read + (1 - hit_rate) x write. With a 1.25x write price, that falls below the uncached price only above a 21.7% hit rate. With Anthropic's 2x one-hour write price, the threshold is 52.6%.

The same formula reproduces Deriv's result: an 85.8% hit rate on a model with no write charge gives exactly 77.2% lower input cost. It also shows why the move to write premiums matters. Deriv's "poorly structured" baseline of about 20% still saved money when writes were free; at a 1.25x write price the same layout would cost more than no caching.

This calculation must use realized hits. Include storage fees, keepalive requests, regional routing effects, and the cost of misses caused by prompt drift. If a workload sits below its break-even rate for a week, remove its explicit breakpoints until the prompt layout is fixed.

How do provider caching contracts differ in 2026?

We reviewed the current prompt-caching and rate-limit documentation from OpenAI, Anthropic, Google, Amazon Bedrock, Groq, and xAI. All document a stable-prefix rule and expose cache-use metrics, but their lifetimes, minimum sizes, write charges, storage treatment, and rate-limit behavior differ.

The table is a planning snapshot, not a permanent configuration reference. Provider contracts change. Keep model-specific values in configuration, link each value to its source, and add an automated check when a model or API version changes.

The rate-limit column matters more than it looks. Anthropic's own example shows a 2 million input-tokens-per-minute limit handling 10 million total input tokens a minute at an 80% hit rate. A gateway that assumes the wrong rule wastes that headroom: LiteLLM's proxy counted cached tokens toward per-customer limits, throttling users five to ten times earlier than the providers would.

What does the same cached token cost across providers?

For agent workloads, most input is read from cache, so the cached price matters more than the headline input price. We analyzed the 61 provider and model listings in the Artificial Analysis prompt caching table.

Method: discount equals one minus the cache-hit price divided by the standard input price, as listed on that page. Storage fees and write charges are billed separately by some providers and are not included.

What did the documentation audit reveal?

The stable-prefix recommendation is now consistent across providers. The surrounding economics are not. Teams can standardize prompt construction and telemetry, but TTL selection, write admission, storage accounting, and rate-limit planning must remain provider-aware.

That distinction is useful in a multi-model stack. A common request envelope can carry stable_prefix_id, prompt_version, expected_reuse_window, and tenant_scope. A provider adapter can translate those fields into supported controls without pretending every cache has the same lifecycle.

What breaks LLM caches in production?

Most low hit rates come from prompt drift. Most dangerous hits come from weak scoping or invalidation. Review both sides of the problem.

Dynamic metadata in the first prompt block

Timestamps, request IDs, user names, and rotating examples near the beginning move the first changed token forward. Place them at the tail or exclude them from model-visible content when they add no value.

Unstable serialization

Semantically identical JSON can serialize differently because of key order, whitespace, float formatting, or omitted defaults. Anthropic's docs call out that some languages, Swift and Go among them, randomize key order during JSON conversion. Canonicalize before hashing and test the exact serialized bytes in continuous integration.

Tool schemas that change order

An agent may expose the same tools in a different order after a process restart or plugin refresh. Sort tools by a durable identifier and version the schema. Deriv hit this with skill blocks rendered in load order until it sorted them by name.

Writes that never become reads

Automatic caching can hide a write-heavy workload. Track read-to-write ratio by prefix. Admit an explicit cache only when predicted reuse beats the write premium and storage cost. Parallel fan-out makes this worse: Anthropic states that a cache entry is available only after the first response begins, so send one request, wait for its first token, then send the rest.

Lifetime changes you did not make

Client libraries and agents choose the cache lifetime for you. Claude Code issue #46829 documented a shift from 1-hour to 5-minute writes in March 2026, and issue #84253 reported that version 2.1.218 stopped requesting the 1-hour lifetime. Alert on the lifetime mix, not only on total tokens.

Mid-session changes that restart the cache

Switching model, changing reasoning effort, or resuming a session after the cache expired all force a cold start. The VS Code team plans to flag these actions in the product so users can decide before paying for a rewrite. Make these changes at task boundaries.

Cache keys without authorization context

A key built from query text alone can return data from another tenant or from a permission scope the user no longer has. Bind identity and access policy before any exact or semantic lookup. Permission revocation must invalidate relevant entries immediately.

Semantic similarity treated as correctness

Two questions can be close in embedding space and require different answers. "Can I delete this workspace?" and "Can an admin delete this workspace?" may retrieve the same candidate but have different authorization rules. Use task-specific thresholds, negative test sets, and a verifier for borderline matches.

Region and model changes hidden behind a gateway

Cross-region routing can reduce locality and create new cache writes. A model revision can change tokenization or invalidate provider-side entries. Record the resolved region and model revision in telemetry even when the application calls a stable gateway endpoint.

Stale entries that look healthy

Availability metrics do not detect a confident answer generated from an old policy or document. Add freshness and provenance fields to cached values. Evaluate invalidation with the same rigor as retrieval quality: update a source, revoke access, rotate a policy, and confirm the next request cannot reuse the old entry.

How should you choose a cache strategy?

Start with prefix stability and exact reuse, because both are observable and reversible. Add semantic reuse later, because its errors can look plausible and evade standard uptime monitoring.

How does ML.ai fit into an LLM caching stack?

Caching removes work you already paid for once. ML.ai lowers the cost of the work that still has to run, and keeps coding agents in the shape that caches well.

ML.ai Inference: route the misses, keep the warm prefix

ML.ai Inference puts one endpoint in front of more than 40 models and sends each request to the most cost-efficient model that still clears your quality bar. It runs in a loop you can inspect: it shadows your existing frontier calls with no user impact, fine-tunes task models on that traffic, shifts load only after your evals pass, and watches for drift with rollback per workload.

Routing and caching pull against each other, because switching models throws away the warm prefix. The answer is to route at task boundaries, not on every turn. Our guide to model routing shows why a warm cache read on a frontier model can cost less than an uncached read on a cheaper one.

ML.ai Code: fewer, shorter prefixes in the editor

ML.ai Code is a coding agent for VS Code and Cursor. Several of its controls map directly to the strategies above:

What ML.ai has delivered

NeoSapien runs a voice-first consumer assistant with five pipeline stages. After ML.ai took over the routine stages and left the hardest ones on frontier models, NeoSapien reported a 42% lower monthly AI bill, 28% faster responses and zero quality regressions, with full rollout in 21 days.

Across deployments, ML.ai designs for 30% to 45% lower cost and 15% to 40% lower latency with quality at or above parity; those are design targets, not averages, and the NeoSapien result reflects the full stack rather than caching alone.

Where should you start with LLM caching this week?

Add cache-read, cache-write and lifetime fields to your request logs, compute hit rate per request class, and look at the lowest one. If it sits under 22% with write-priced caching, fix the prompt order before anything else. Then place breakpoints at your most stable boundaries, and only after that consider semantic reuse with bypass rules and a verifier.

If your agents are the expensive part, ML.ai Code keeps sessions short and cache-friendly from the editor, and a 30-day ML.ai Inference pilot shows what routing and caching save together on your own traffic.

Frequently Asked Questions

What is the difference between prompt caching and response caching?

Prompt caching reuses provider-side computation for an identical prefix, then generates a fresh answer. Response caching returns a previously generated answer. Prompt caching has lower correctness risk because generation still runs; response caching saves more work but requires stricter keys and invalidation.

Does prompt caching change the model's answer?

No. Anthropic states that caching has no effect on output generation. The same prompt, parameters, and model behave as they would without the cache. Application and semantic caches can change behavior because they may return a stored answer.

What cache hit rate should an AI agent target?

Above 22% just to break even when writes cost 1.25x, and well above that to matter. Well-structured agents reach 85% to 94%: Deriv reports 85.8% and the VS Code team about 94%. Independent tests across 500 agent sessions measured 41% to 80% cost savings.

Do cached tokens count against rate limits?

It depends on the provider. OpenAI and xAI count cached input toward tokens-per-minute limits. Anthropic (except Claude Haiku 3.5), Amazon Bedrock and Groq do not. Treat this as a provider-specific contract and make sure your gateway applies the same rule.

When is semantic caching unsafe?

It is unsafe when small wording differences can change authorization, time, geography, customer identity, medical or legal meaning, or live data. Route code and time-sensitive queries around the cache, as Walmart's decision engine does, and use exact caching or fresh generation unless a verifier can enforce the relevant constraints.

How often should an LLM cache be invalidated?

Invalidate on the earliest event that can change correctness: source updates, deletes, permission changes, prompt or tool-schema versions, model revisions, policy changes, or TTL expiry. Event-based invalidation should complement time-based expiry.