Info-Aware Routing
Send every request to the best provider right now.
Info-Aware Routing lets Agent Router (EAIG) choose a provider and model per request, using live cost, latency, health, region, and request metadata, and fail over automatically when a provider degrades.
What Info-Aware Routing does
The plugin maintains a live view of every configured provider and model: health, observed latency, error rate, price per token, and region. For each request, a routing decision engine filters out providers that are not eligible, scores the remaining candidates using your routing policy, and sends the request to the best one, with ordered fallbacks if it fails.
Static routing to a single provider means paying premium prices for simple requests, absorbing that provider's latency spikes, and suffering full outages when it goes down. Teams that hand-code failover in each application end up with inconsistent, brittle logic.
Info-Aware Routing runs in Agent Router after authentication, quota, and guardrail checks, at the point where a backend is selected. Applications call one endpoint, and the gateway decides where each request goes.
The request and response flow
- 1Request
Read request context
Agent Router reads the requested model family, priority, tenant, region, data-classification tag, and any routing headers.
- 2Control
Filter eligible providers
Providers are removed if they are unhealthy, out of the allowed region, not approved for the data classification, or over their capacity.
- 3Control
Score candidates
Remaining providers are scored on cost, recent p95 latency, and error rate using the weights in the routing policy.
- 4Request
Route and fail over
The request goes to the highest-scoring provider. On timeouts or 5xx errors, Agent Router retries the next candidate automatically.
- 5Response
Feed back live signals
Response latency, errors, and token cost are recorded and update each provider's health and latency scores for the next decision.
Info-Aware Routing in Agent Router
The decision engine filters and scores providers on live signals, then routes each request to the best candidate.
Built for production AI traffic
Cost-based routing
Prefer the lowest price per token among providers that meet quality and latency requirements.
Latency-based routing
Use observed p95 latency to route interactive traffic to the fastest healthy provider.
Health-aware failover
Detect errors and timeouts, eject unhealthy providers, and retry the next candidate automatically.
Region and data residency
Keep requests within allowed regions and send sensitive data only to approved or self-hosted models.
Metadata and priority routing
Route by tenant, priority, or data-classification headers, for example premium traffic to dedicated capacity.
Weighted load distribution
Spread traffic across providers by weight to use committed capacity and avoid rate limits.
Engineering and business outcomes
Survive provider outages
Automatic failover keeps applications running when a provider or region degrades.
Pay less for routine requests
Simple, high-volume traffic goes to cost-efficient providers while demanding requests get premium models.
Faster interactive responses
Live latency scoring steers user-facing traffic away from slow providers.
Residency by policy
Region and data-classification rules are enforced on every request, not left to each application.
How the decision engine chooses
Concrete examples of the conditions Agent Router evaluates and the routing decision it makes.
| Scenario | Conditions observed | Decision |
|---|---|---|
| Normal operation | Provider A healthy, p95 latency 850 ms, lowest cost per token | Route to Provider A |
| Provider A degrades | Provider A error rate above threshold, health check failing | Eject A, fail over to Provider B |
| Latency spike | Provider A p95 rises above the 2 s latency budget for interactive traffic | Route interactive requests to Provider B, keep batch on A |
| EU customer request | Request tagged region=eu, data residency policy applies | Route to Provider C (EU region) only |
| Confidential data | Request tagged data-class=confidential | Route to Provider D (self-hosted) only |
| Batch job, low priority | Header priority=low, cost weight dominant | Route to cheapest healthy provider |
| All external providers down | A, B, and C failing health checks | Serve from Provider D, return degraded-mode header |
Where enterprises use Info-Aware Routing
Multi-provider resilience
Run OpenAI, Azure OpenAI, Anthropic, and Bedrock side by side and fail over between them automatically.
Cost-optimized batch processing
Send overnight summarization and classification jobs to the lowest-cost healthy provider.
Data residency for global products
Keep EU customer prompts on EU-hosted models and US traffic on US regions.
Tiered service levels
Route premium-tier customers to dedicated low-latency capacity and free-tier traffic to shared pools.
Hybrid cloud and self-hosted models
Send confidential workloads to vLLM on your own GPUs while general traffic uses cloud providers.
Cost, latency, and health-aware routing policy
- Candidate providersProviders and models that can serve the route.
- A (OpenAI), B (Azure OpenAI), C (Bedrock EU), D (self-hosted vLLM)
- Eligibility filtersRules that remove providers before scoring, for example unhealthy or out-of-region ones.
- Health, region, data classification
- Scoring weightsHow strongly each signal counts when ranking eligible providers.
- Cost 50% · latency 35% · error rate 15%
- Latency budgetProviders slower than this are deprioritized for interactive requests.
- 2 seconds (p95) for interactive traffic
- Failover triggersResponses that cause Agent Router to retry the next provider.
- 5xx errors, timeouts, HTTP 429
- Maximum attemptsHow many providers are tried before the request fails.
- 3
- Outlier ejectionWhen an unhealthy provider is temporarily taken out of rotation.
- 5 consecutive errors → remove for 30 s
Illustrative example. Settings and values show what can be configured and are not exact configuration syntax.
How it fits your AI platform
- 01
Platform team
Registers providers and models and defines routing weights, filters, and failover rules.
- 02
Applications
Call one Agent Router endpoint and optionally pass priority, region, or data-class headers.
- 03
Agent Router
Evaluates live signals and request metadata and routes each request to the best provider.
- 04
Models & providers
Receive traffic according to health, cost, and policy, with failover when one degrades.
- 05
SRE & FinOps
Monitor routing decisions, failovers, latency, and cost per provider from gateway metrics.
Route every AI request intelligently
IMESH can help you design routing policies, failover strategies, and multi-provider architectures on Agent Router (EAIG).