What is an LLM cache and where does it sit in an AI application?

Application response caches

An exact response cache returns a stored result when its key matches. A semantic cache can reuse a result for a sufficiently similar request. Both sit in the application layer and need rules for identity, permissions, freshness and invalidation so reuse does not cross boundaries that matter to correctness.

Provider prompt caching

Some model providers can treat repeated prompt prefixes as cached input under model-specific rules. The provider still receives a request, but usage and pricing may distinguish cached from uncached input. Stable prompt construction often matters for achieving cache hits.

Measure the mechanism you actually use

Track cache hit rate, latency, cached and uncached provider usage, incorrect reuse and freshness. Do not combine different cache types into one headline number because a response cache hit and a provider cached-input token represent different events.

Measurement checklist

1. Choose a representative completed task, not an artificial one-line prompt.
2. Record the selected model, provider input, cached input, output, retries and final result.
3. Change one optimization mechanism at a time so the cause remains visible.
4. Verify required identifiers, tool calls, code changes or business fields.
5. Keep passthrough available when the reduced request does not pass.

Varion commercial evaluation

Varion keeps proprietary product implementation details private. Evaluate Token Optimisation on representative traffic and judge it by the measured commercial result. New verified users receive 100,000 processed input tokens and 50 local test runs.