Exact response cache
Exact caching is safest when the request is deterministic, permissions are isolated and the response remains valid. Cache keys must include every input that changes behavior. Tenant isolation and expiration rules are essential.
Provider prompt cache
Prompt caching is controlled by the model provider and reported through provider usage. It can help large stable prefixes without returning an old answer. Eligibility and pricing vary by model.
Local token estimate
This is an approximate comparison, not provider billing data.
Semantic cache
Semantic caching can save the full model call, but similarity is not equivalence. It needs strict thresholds, domain checks and a strategy for rapidly changing data. Varion currently emphasizes exact cache and measured context reduction rather than presenting semantic similarity as guaranteed correctness.
Measurement checklist
- Choose a representative completed task, not an artificial one-line prompt.
- Record the selected model, provider input, cached input, output, retries and final result.
- Change one optimization mechanism at a time so the cause remains visible.
- Verify required identifiers, tool calls, code changes or business fields.
- Keep passthrough available when the reduced request does not pass.
How Varion fits
Varion Token Engine is a gateway and testing platform for reducing eligible input-token waste across supported AI traffic. It reports original and provider-bound input, keeps provider charges separate, and does not claim that every request can be reduced. New verified users receive 100,000 processed input tokens and 50 local test runs.
Frequently asked questions
Which cache is safest?
Exact caching of a fully identical, permission-safe request is easier to reason about than semantic reuse, but every system still needs invalidation and isolation.
Does an LLM cache reduce input tokens?
An exact response cache can avoid the provider call. Prompt caching changes provider processing or price. Context reduction sends fewer tokens.