Semantic Caching
Answer repeat questions from cache, not from the model.
Semantic Caching lets Agent Router (EAIG) recognize when a new prompt means the same thing as one it has already answered, and return the stored response in milliseconds instead of paying for another model call.
What Semantic Caching does
The plugin converts each incoming prompt into an embedding and compares it with prompts that have already been answered. When a stored prompt is similar enough, Agent Router returns the cached response. When nothing matches, the request continues to the model and the new response is stored for next time.
Enterprise AI traffic is highly repetitive: support assistants, internal copilots, and RAG applications receive the same questions phrased in many different ways. Exact-match caches miss these, so every variation becomes a full, billable model call with full model latency.
Semantic Caching runs inside Agent Router, after authentication and before routing to a provider. Because it sits at the gateway, every application behind Agent Router benefits without changing application code or SDKs.
The request and response flow
- 1Request
Request arrives at Agent Router
An application sends a chat or completion request. Agent Router authenticates it and identifies the route, model, and tenant.
- 2Request
Prompt is embedded
The plugin normalizes the prompt and generates an embedding using the configured embedding model.
- 3Request
Similarity evaluation
The embedding is compared against cached entries scoped to the same route, model, and tenant. The closest match is scored against the similarity threshold.
- 4Response
Cache hit: respond immediately
If the score meets the threshold and the entry is within its TTL, the cached response is returned with a cache-status header. No provider call is made.
- 5Response
Cache miss: call the model and store
Otherwise the request is routed to the LLM provider as normal. The response is returned to the application and written to the cache with its embedding and TTL.
Semantic Caching in Agent Router
Similar prompts are answered from the cache. Misses go to the provider and are cached on the way back.
Built for production AI traffic
Embedding-based matching
Matches prompts by meaning rather than exact text, so rephrased questions still hit the cache.
Tunable similarity thresholds
Set how close a match must be per route, from strict thresholds for factual answers to looser ones for FAQs.
TTL and invalidation
Expire entries by age, and clear them by route or tenant when source content changes.
Scoped cache keys
Entries are isolated by tenant, route, and model so one team never receives another team's cached answer.
Selective bypass
Skip caching for streaming, tool calls, or requests carrying a bypass header.
Cache observability
Hit rate, latency saved, and tokens avoided are exported as gateway metrics for dashboards and FinOps.
Engineering and business outcomes
Fewer billable model calls
Every cache hit is a request that consumes no input or output tokens at the provider. Savings grow with how repetitive your traffic is.
Millisecond responses on hits
Cache hits skip model inference entirely, turning multi-second completions into near-instant responses.
Less exposure to provider limits
Fewer upstream calls means less pressure on provider rate limits and quotas during traffic spikes.
No application changes
Caching is enabled per route at the gateway. Teams get the benefit without adding cache logic to every service.
Where enterprises use Semantic Caching
Customer support assistants
Thousands of customers ask about the same refund, shipping, or password-reset policies in different words.
Internal knowledge copilots
Employees repeatedly ask about HR policies, onboarding steps, and engineering standards.
RAG over stable documentation
Product documentation and runbooks change rarely, so answers can be reused safely within a TTL.
Classification and tagging pipelines
Batch jobs that classify similar tickets, emails, or log lines avoid re-sending near-duplicate prompts.
Enable semantic caching on a route
- Embedding modelModel used to turn each prompt into an embedding for comparison.
- text-embedding-3-small
- Similarity thresholdHow close a new prompt must be to a cached one to count as a hit. Higher is stricter.
- 0.92 (cosine)
- Cache TTLHow long a cached response can be reused before it expires.
- 6 hours
- Cache scopeKeeps cached answers separate so one team never receives another team's response.
- Tenant · route · model
- Maximum entriesUpper bound on stored responses before the oldest are evicted.
- 50,000 per route
- Bypass rulesRequests that always skip the cache and go straight to the model.
- Streaming, tool calls, x-cache-bypass header
Illustrative example. Settings and values show what can be configured and are not exact configuration syntax.
How it fits your AI platform
- 01
Platform team
Enables Semantic Caching on selected Agent Router routes and sets thresholds and TTLs per use case.
- 02
Applications
Call Agent Router exactly as before, using the same OpenAI-compatible API.
- 03
Agent Router
Authenticates the request, runs the cache lookup, and serves hits directly.
- 04
Models & providers
Receive only cache misses, so paid inference is reserved for genuinely new questions.
- 05
FinOps & SRE
Track hit rate, tokens avoided, and latency saved from gateway metrics.
Cut LLM cost and latency with Semantic Caching
IMESH can help you identify cacheable traffic, tune similarity thresholds, and roll out Semantic Caching on Agent Router (EAIG).