How Model Aware Traffic Routing Supports An Effective AI Cost Management Strategy

July 8, 2026

Eitan Suez

AI cost management starts with smarter routing. See how Virtual Models in agentgateway 1.3.0 abstract provider models and use intelligent routing policies to balance cost, quality, and reliability across LLM providers.

The agentgateway project has supported what is called "model-aware" traffic routing for some time. In his blog Body-Based Routing in Agentgateway, John Howard describes how agentgateway implements body-based routing in a way that exhibits 4x to 5x improvement in throughput and latency over the traditional mechanism using a custom external processor. The key is the definition of a custom pre-routing expression to parse the model name from the body of the request, and to expose it as a header, which can subsequently be used to implement routing decisions.

Fast forward to today, and agentgateway's latest release, which provides comprehensive support for AI cost management.

In this blog, we explore a new feature in version 1.3.0, named Virtual Models, and specifically how Virtual Models supports an effective cost management strategy.

What are virtual models?

Essentially, Virtual Models are about insulating users from provider-specific model names, which can change over time, or get deprecated. Better yet, it provides an effective way to map semantically meaningful model names (that clearly communicate intent) from the actual backing provider model names, which ultimately change over time.

For example, one could define virtual models such as "fast", or "reasoning" that imply a particular desired model attribute. These virtual model names map to suitable provider models that exhibit those traits. Over time, the mapping can be updated and improved without affecting users who continue to use the virtual model names.

Virtual Model mapping does not have to be one-to-one: the Virtual models feature offers three routing strategies (with perhaps more to come): weighted, failover, and conditional. These strategies represent intelligent ways of expressing which model to route to according to a specified algorithm.

Let's briefly review each in turn.

Weighted routing

A simple strategy whereby weights are assigned to different models.

This can be useful for a variety of purposes such as A/B testing: 50% to gpt-4o and 50% to gpt-4o-mini can be a way to A/B-test a currently used model against a speedy and cheaper variant. This routing strategy can also be useful for "canary-releasing" a new model. The canary approach can lessen the risk of migrating to a new model by not impacting the entire population of users: 90% for the current or primary model and 10% to the new canary model, then gradually shifting the weights to the new "current" model.

The documentation provides a 90/10 weighted routing example which you can review here.

llm:
  models:
  - name: gpt-4o-public
    visibility: public
    provider: openAI
    params:
      model: gpt-4o
      apiKey: "$OPENAI_API_KEY"
  - name: gpt-4o-primary
    visibility: internal
    provider: openAI
    params:
      model: gpt-4o
      apiKey: "$OPENAI_API_KEY"
  - name: gpt-4o-fallback
    visibility: internal
    provider: openAI
    params:
      model: gpt-4o-mini
      apiKey: "$OPENAI_API_KEY"

  virtualModels:
  - name: smart
    routing:
      weighted:
        targets:
        - model: gpt-4o-primary
          weight: 90
        - model: gpt-4o-fallback
          weight: 10

Failover routing

This strategy allows for the definition of fallback or failover models that kick in when the primary one is unavailable, or when a user or team has reached their quota or were rate-limited. Fallback models can be selected in order of decreasing costs, i.e. to failover to cheaper models to continue to provide some level of service.

Here is the example that the documentation provides for failover routing:

llm:
  models:
  - name: claude-primary
    visibility: internal
    provider: anthropic
    params:
      model: claude-sonnet-4-0
      apiKey: "$ANTHROPIC_API_KEY"
  - name: claude-backup-a
    visibility: internal
    provider: anthropic
    params:
      model: claude-3-5-haiku-20241022
      apiKey: "$ANTHROPIC_API_KEY"
  - name: claude-backup-b
    visibility: internal
    provider: anthropic
    params:
      model: claude-3-5-haiku-20241022
      apiKey: "$ANTHROPIC_API_KEY"

  virtualModels:
  - name: resilient
    routing:
      failover:
        targets:
        - model: claude-primary
          priority: 1
        - model: claude-backup-a
          priority: 2
        - model: claude-backup-b
          priority: 2

Conditional routing

The conditional routing strategy lets us define custom routing rules that are a function of organization-defined attributes. The documentation provides an example, where distinct tiers of users are represented through a request header named "x-tier". where "pro" users are routed to a smarter and more lucrative model, whereas "free" tier users get sent to the cheaper "fast" model:

llm:
  models:
  - name: openai-public
    visibility: public
    provider: openAI
    params:
      model: gpt-4o-mini
      apiKey: "$OPENAI_API_KEY"
  - name: openai-fast
    visibility: internal
    provider: openAI
    params:
      model: gpt-4o-mini
      apiKey: "$OPENAI_API_KEY"
  - name: openai-smart
    visibility: internal
    provider: openAI
    params:
      model: gpt-4o
      apiKey: "$OPENAI_API_KEY"

  virtualModels:
  - name: adaptive
    routing:
      conditional:
        targets:
        - model: openai-fast
          when: request.headers["x-tier"] == "free"
        - model: openai-smart
          when: request.headers["x-tier"] == "pro"

Summary

One theme of the v1.3.0 release of agentgateway is helping end users with managing AI costs, and Virtual Models is one of several of these new and exciting features.

Virtual Models provide several configurable strategies for routing requests to the "right" model - the one that does the job, while at the same time keeping costs low.

Look out for continued evolution of this feature in subsequent versions of agentgateway, which could even further simplify our job as administrators of configuring practical and cost-sensible routing rules to backend models.