← All Plugins
AI Routing

Info-Aware Routing

Send every request to the right model for its complexity and conditions.

Info-Aware Routing lets Agent Router (EAIG) score each prompt's complexity and route it to the model that fits, using live cost, latency, health, and request metadata, and fall back automatically when a model degrades.

What Info-Aware Routing does

What it does

For each request, the plugin calculates a prompt complexity score and compares it with the complexity range each model handles well, for example Llama 3.1 8B for simple prompts, Qwen3-32B for medium ones, and Claude Sonnet 5 for complex ones. It also checks live health, latency, cost, region, and request metadata, then routes to the best eligible model with an ordered fallback.

The enterprise problem

Static routing to a single provider means paying premium prices for simple requests, absorbing that provider's latency spikes, and suffering full outages when it goes down. Teams that hand-code failover in each application end up with inconsistent, brittle logic.

Where it fits in Agent Router

Info-Aware Routing runs in Agent Router after authentication, quota, and guardrail checks, at the point where a backend is selected. Applications call one endpoint, and the gateway decides where each request goes.

How It Works

The request and response flow

  1. 1
    Request

    Score the prompt

    Agent Router calculates a prompt complexity score and reads priority, region, data-classification, and routing headers.

  2. 2
    Control

    Filter eligible models

    Models are removed if they are unhealthy, out of the allowed region, not approved for the data classification, or over capacity.

  3. 3
    Control

    Match and rank

    Eligible models are matched on prompt complexity fit, then ranked on cost, recent p95 latency, and error rate.

  4. 4
    Request

    Route and fall back

    A score of 0.62 routes to Qwen3-32B. On timeouts or 5xx errors, Agent Router falls back to the next model, such as Claude Sonnet 5.

  5. 5
    Response

    Feed back live signals

    Response latency, errors, and token cost are recorded and update each model's health and latency scores for the next decision.

Architecture

Info-Aware Routing in Agent Router

Info-Aware Routing architecture: Prompt, priority=high, region=us, data-class=public send requests to Agent Router, which applies Prompt complexity score, then Routing decision engine, then Health · latency · cost, and connects to Llama 3.1 8B, Qwen3-32B, Claude Sonnet 5.ApplicationOne endpointRequest metadatapriority=highregion=usdata-class=publicAgent RouterInfo-Aware RoutingPrompt complexity scoreSimpleComplex0.62ModelHealthp95CostComplexity fitLlama 3.1 8B420 ms$0.0 – 0.4Qwen3-32B850 ms$$0.4 – 0.7Claude Sonnet 51.4 s$$$0.7 – 1.0Score 0.62 fits Qwen3-32B → route there, fall back to Claude Sonnet 5Llama 3.1 8BSimple prompts · vLLMQwen3-32BSelected · vLLM on-premClaude Sonnet 5Fallback · AWS BedrockRoute
Applications
Promptpriority=highregion=usdata-class=public
Agent Router
Info-Aware Routing plugin
Prompt complexity score
Routing decision engine
Health · latency · cost
Models
Llama 3.1 8BSimple prompts · vLLM
Qwen3-32BSelected · vLLM on-prem
Claude Sonnet 5Fallback · AWS Bedrock

The decision engine scores the prompt, matches it to the model that fits, checks live signals, and routes with a fallback.

Key Features

Built for production AI traffic

Cost-based routing

Prefer the lowest price per token among providers that meet quality and latency requirements.

Latency-based routing

Use observed p95 latency to route interactive traffic to the fastest healthy provider.

Health-aware failover

Detect errors and timeouts, eject unhealthy providers, and retry the next candidate automatically.

Region and data residency

Keep requests within allowed regions and send sensitive data only to approved or self-hosted models.

Metadata and priority routing

Route by tenant, priority, or data-classification headers, for example premium traffic to dedicated capacity.

Prompt complexity routing

Score each prompt and send simple ones to small, fast models such as Llama 3.1 8B and complex ones to Claude Sonnet 5 or GPT-4.1.

Benefits

Engineering and business outcomes

Reliability

Survive provider outages

Automatic failover keeps applications running when a provider or region degrades.

Cost

Pay less for routine requests

Simple, high-volume traffic goes to cost-efficient providers while demanding requests get premium models.

Latency

Faster interactive responses

Live latency scoring steers user-facing traffic away from slow providers.

Compliance

Residency by policy

Region and data-classification rules are enforced on every request, not left to each application.

Routing Decisions

How the decision engine chooses

Concrete examples of the conditions Agent Router evaluates and the routing decision it makes.

ScenarioConditions observedDecision
Simple promptPrompt complexity 0.21, Llama 3.1 8B healthyRoute to Llama 3.1 8B
Medium promptPrompt complexity 0.62, Qwen3-32B healthy, p95 850 msRoute to Qwen3-32B
Complex promptPrompt complexity 0.88Route to Claude Sonnet 5
Qwen3-32B degradesError rate above threshold, health check failingFall back to Claude Sonnet 5
Latency spikeQwen3-32B p95 above the 2 s budget for interactive trafficRoute interactive requests to the next fastest model
Confidential dataRequest tagged data-class=confidentialSelf-hosted Llama or Qwen models on vLLM only
Batch job, low priorityHeader priority=low, cost weight dominantRoute to the cheapest healthy model
Use Cases

Where enterprises use Info-Aware Routing

01

Multi-model resilience

Run Llama and Qwen on vLLM alongside Claude Sonnet 5 on AWS Bedrock and GPT-4.1 on Azure OpenAI, and fall back between them automatically.

02

Cost-optimized batch processing

Send overnight summarization and classification jobs to the lowest-cost healthy provider.

03

Data residency for global products

Keep EU customer prompts on EU-hosted models and US traffic on US regions.

04

Tiered service levels

Route premium-tier customers to dedicated low-latency capacity and free-tier traffic to shared pools.

05

Hybrid cloud and self-hosted models

Send confidential workloads to vLLM on your own GPUs while general traffic uses cloud providers.

Configuration

Cost, latency, and health-aware routing policy

Candidate modelsModels that can serve the route.
Llama 3.1 8B, Qwen3-32B (vLLM) · Claude Sonnet 5 (AWS Bedrock) · GPT-4.1 (Azure OpenAI)
Eligibility filtersRules that remove providers before scoring, for example unhealthy or out-of-region ones.
Health, region, data classification
Complexity rangesWhich prompt complexity scores each model handles.
Llama 3.1 8B 0–0.4 · Qwen3-32B 0.4–0.7 · Claude Sonnet 5 0.7–1.0
Scoring weightsHow strongly each signal counts when ranking eligible models.
Cost 50% · latency 35% · error rate 15%
Latency budgetProviders slower than this are deprioritized for interactive requests.
2 seconds (p95) for interactive traffic
Failover triggersResponses that cause Agent Router to retry the next provider.
5xx errors, timeouts, HTTP 429
Outlier ejectionWhen an unhealthy provider is temporarily taken out of rotation.
5 consecutive errors → remove for 30 s

Illustrative example. Settings and values show what can be configured and are not exact configuration syntax.

Enterprise Workflow

How it fits your AI platform

  1. 01

    Platform team

    Registers providers and models and defines routing weights, filters, and failover rules.

  2. 02

    Applications

    Call one Agent Router endpoint and optionally pass priority, region, or data-class headers.

  3. 03

    Agent Router

    Evaluates live signals and request metadata and routes each request to the best provider.

  4. 04

    Models & providers

    Receive traffic according to health, cost, and policy, with failover when one degrades.

  5. 05

    SRE & FinOps

    Monitor routing decisions, failovers, latency, and cost per provider from gateway metrics.

Route every AI request intelligently

IMESH can help you design routing policies, failover strategies, and multi-provider architectures on Agent Router (EAIG).