Info-Aware Routing
Send every request to the right model for its complexity and conditions.
Info-Aware Routing lets Agent Router (EAIG) score each prompt's complexity and route it to the model that fits, using live cost, latency, health, and request metadata, and fall back automatically when a model degrades.
What Info-Aware Routing does
For each request, the plugin calculates a prompt complexity score and compares it with the complexity range each model handles well, for example Llama 3.1 8B for simple prompts, Qwen3-32B for medium ones, and Claude Sonnet 5 for complex ones. It also checks live health, latency, cost, region, and request metadata, then routes to the best eligible model with an ordered fallback.
Static routing to a single provider means paying premium prices for simple requests, absorbing that provider's latency spikes, and suffering full outages when it goes down. Teams that hand-code failover in each application end up with inconsistent, brittle logic.
Info-Aware Routing runs in Agent Router after authentication, quota, and guardrail checks, at the point where a backend is selected. Applications call one endpoint, and the gateway decides where each request goes.
The request and response flow
- 1Request
Score the prompt
Agent Router calculates a prompt complexity score and reads priority, region, data-classification, and routing headers.
- 2Control
Filter eligible models
Models are removed if they are unhealthy, out of the allowed region, not approved for the data classification, or over capacity.
- 3Control
Match and rank
Eligible models are matched on prompt complexity fit, then ranked on cost, recent p95 latency, and error rate.
- 4Request
Route and fall back
A score of 0.62 routes to Qwen3-32B. On timeouts or 5xx errors, Agent Router falls back to the next model, such as Claude Sonnet 5.
- 5Response
Feed back live signals
Response latency, errors, and token cost are recorded and update each model's health and latency scores for the next decision.
Info-Aware Routing in Agent Router
The decision engine scores the prompt, matches it to the model that fits, checks live signals, and routes with a fallback.
Built for production AI traffic
Cost-based routing
Prefer the lowest price per token among providers that meet quality and latency requirements.
Latency-based routing
Use observed p95 latency to route interactive traffic to the fastest healthy provider.
Health-aware failover
Detect errors and timeouts, eject unhealthy providers, and retry the next candidate automatically.
Region and data residency
Keep requests within allowed regions and send sensitive data only to approved or self-hosted models.
Metadata and priority routing
Route by tenant, priority, or data-classification headers, for example premium traffic to dedicated capacity.
Prompt complexity routing
Score each prompt and send simple ones to small, fast models such as Llama 3.1 8B and complex ones to Claude Sonnet 5 or GPT-4.1.
Engineering and business outcomes
Survive provider outages
Automatic failover keeps applications running when a provider or region degrades.
Pay less for routine requests
Simple, high-volume traffic goes to cost-efficient providers while demanding requests get premium models.
Faster interactive responses
Live latency scoring steers user-facing traffic away from slow providers.
Residency by policy
Region and data-classification rules are enforced on every request, not left to each application.
How the decision engine chooses
Concrete examples of the conditions Agent Router evaluates and the routing decision it makes.
| Scenario | Conditions observed | Decision |
|---|---|---|
| Simple prompt | Prompt complexity 0.21, Llama 3.1 8B healthy | Route to Llama 3.1 8B |
| Medium prompt | Prompt complexity 0.62, Qwen3-32B healthy, p95 850 ms | Route to Qwen3-32B |
| Complex prompt | Prompt complexity 0.88 | Route to Claude Sonnet 5 |
| Qwen3-32B degrades | Error rate above threshold, health check failing | Fall back to Claude Sonnet 5 |
| Latency spike | Qwen3-32B p95 above the 2 s budget for interactive traffic | Route interactive requests to the next fastest model |
| Confidential data | Request tagged data-class=confidential | Self-hosted Llama or Qwen models on vLLM only |
| Batch job, low priority | Header priority=low, cost weight dominant | Route to the cheapest healthy model |
Where enterprises use Info-Aware Routing
Multi-model resilience
Run Llama and Qwen on vLLM alongside Claude Sonnet 5 on AWS Bedrock and GPT-4.1 on Azure OpenAI, and fall back between them automatically.
Cost-optimized batch processing
Send overnight summarization and classification jobs to the lowest-cost healthy provider.
Data residency for global products
Keep EU customer prompts on EU-hosted models and US traffic on US regions.
Tiered service levels
Route premium-tier customers to dedicated low-latency capacity and free-tier traffic to shared pools.
Hybrid cloud and self-hosted models
Send confidential workloads to vLLM on your own GPUs while general traffic uses cloud providers.
Cost, latency, and health-aware routing policy
- Candidate modelsModels that can serve the route.
- Llama 3.1 8B, Qwen3-32B (vLLM) · Claude Sonnet 5 (AWS Bedrock) · GPT-4.1 (Azure OpenAI)
- Eligibility filtersRules that remove providers before scoring, for example unhealthy or out-of-region ones.
- Health, region, data classification
- Complexity rangesWhich prompt complexity scores each model handles.
- Llama 3.1 8B 0–0.4 · Qwen3-32B 0.4–0.7 · Claude Sonnet 5 0.7–1.0
- Scoring weightsHow strongly each signal counts when ranking eligible models.
- Cost 50% · latency 35% · error rate 15%
- Latency budgetProviders slower than this are deprioritized for interactive requests.
- 2 seconds (p95) for interactive traffic
- Failover triggersResponses that cause Agent Router to retry the next provider.
- 5xx errors, timeouts, HTTP 429
- Outlier ejectionWhen an unhealthy provider is temporarily taken out of rotation.
- 5 consecutive errors → remove for 30 s
Illustrative example. Settings and values show what can be configured and are not exact configuration syntax.
How it fits your AI platform
- 01
Platform team
Registers providers and models and defines routing weights, filters, and failover rules.
- 02
Applications
Call one Agent Router endpoint and optionally pass priority, region, or data-class headers.
- 03
Agent Router
Evaluates live signals and request metadata and routes each request to the best provider.
- 04
Models & providers
Receive traffic according to health, cost, and policy, with failover when one degrades.
- 05
SRE & FinOps
Monitor routing decisions, failovers, latency, and cost per provider from gateway metrics.
Route every AI request intelligently
IMESH can help you design routing policies, failover strategies, and multi-provider architectures on Agent Router (EAIG).
