Most apps hardcode a model name such as gpt-4o-mini. When a cheaper or better model ships, each app team that uses the old model edits code and redeploys. A virtual model breaks that link: the app asks for a name you control, such as smart or cheap, and the AI gateway decides which concrete model serves the request.

What is a virtual model?
A virtual model is a model name that exists only on the gateway. It maps to one or more concrete models, each defined under llm.models with its provider and credentials. Apps send the virtual name in the standard model field, so the OpenAI SDK, LangChain, and coding agents work unchanged.
agentgateway routes a virtual model with one of these strategies:
Prerequisites
The examples use two concrete OpenAI models, gpt-4o-mini and gpt-4.1-mini, added on LLM → Models. See Run an AI gateway without Kubernetes for how to add them.
Each task below shows two ways to make the change: Option A in the agentgateway UI, or Option B by editing config.yaml directly. Both produce the same config, so use whichever you prefer.
Split traffic by weight
Option A: In the UI
- Open LLM → Models and click Add virtual model.
- Fill in the form with the values in the table below.
- Click Save virtual model.
smart appears in the model list as a Virtual model with two weighted targets:

Option B: Edit config.yaml directly
Add a virtualModels list under llm:
llm:
virtualModels:
- name: smart
routing:
weighted:
targets:
- model: gpt-4o-mini
weight: 50
- model: gpt-4.1-mini
weight: 50Try it
for i in 1 2 3 4 5 6; do
curl -s localhost:4000/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"smart","max_tokens":3,"messages":[{"role":"user","content":"hi"}]}' | jq -r .model
doneThe output mixes gpt-4o-mini-… and gpt-4.1-mini-…, the dated versions OpenAI returns. Edit smart in the UI, change the weights to 90 and 10, save, and traffic shifts for all apps within a second.
Route by a request attribute
Conditional routing looks at the request. The tiered virtual model sends free-tier customers to the cheaper model and everyone else to the larger one, based on an x-tier header.
Option A: In the UI
- On LLM → Models, click Add virtual model.
- Fill in the form with the values in the table below.
- Click Save virtual model.
Rule 1's Condition box starts with sample text. Select all of it and replace it.
![The Add virtual model dialog with Conditional routing: rule 1 sends request.headers["x-tier"] == "free" to gpt-4o-mini, and the fallback goes to gpt-4.1-mini.](https://cdn.prod.website-files.com/6704482c45ef6ead081645ff/6afc95f945b4655a6c5a0826_ui-virtual-conditional.png)
Option B: Edit config.yaml directly
llm:
virtualModels:
- name: tiered
routing:
conditional:
targets:
- model: gpt-4o-mini
when: request.headers["x-tier"] == "free"
- model: gpt-4.1-miniEach rule becomes a target with a when field, a CEL expression evaluated on each request. The gateway tries targets in order. The fallback target has no when, so it catches every request the earlier rules didn't match.
Try it
for tier in free pro; do
curl -s localhost:4000/v1/chat/completions -H 'content-type: application/json' -H "x-tier: $tier" \
-d '{"model":"tiered","max_tokens":3,"messages":[{"role":"user","content":"hi"}]}' \
| jq -r --arg t "$tier" '"\($t) -> \(.model)"'
donefree -> gpt-4o-mini-2024-07-18
pro -> gpt-4.1-mini-2025-04-14CEL expressions can read headers, the request body, the caller's JWT claims, and the API key's metadata. For example, "premium" in jwt.groups routes by identity provider group once JWT authentication is on.
Fail over between models
Option A: In the UI
In the Add virtual model dialog, choose Failover as the routing strategy and add the targets.
Option B: Edit config.yaml directly
A failover virtual model groups targets by priority:
llm:
virtualModels:
- name: resilient
routing:
failover:
targets:
- model: gpt-4.1-mini
priority: 0
- model: gpt-4o-mini
priority: 1The gateway prefers the lowest priority value, and within a group it picks the target with the best health and latency score. When every target in a group is degraded, requests move to the next group.
Targets can use different providers, because each concrete model under llm.models has its own provider and credentials. A failover group can put a Bedrock or Anthropic model behind an OpenAI one.
Why not route in the app?
You can write routing logic in each app, but then each app team owns a copy of it. With virtual models:
- The platform team changes which model serves
smartin the UI or in one file, and no app redeploys - A canary for a new model is a weight change, and rolling it back is another
- The gateway logs both the requested name and the concrete model it chose, so cost and latency reports show which model served each request
Frequently asked questions
What is a virtual model in an AI gateway?
A virtual model is a model name defined on the gateway that maps to one or more provider models. Apps request the virtual name, and the gateway picks the concrete model by weight, failover order, a CEL rule, or an external callout.
Do apps need code changes to use virtual models?
No. Apps send the virtual name in the standard model field of an OpenAI-compatible request. The only change from calling the provider directly is the base URL.
Can a virtual model span several LLM providers?
Yes. Each target refers to a model under llm.models, and each of those models has its own provider and credentials, so one virtual model can route between OpenAI, Anthropic, Bedrock, and self-hosted models.
Do authorization rules apply to the virtual model or the concrete model?
To the concrete model the gateway picks. If a user may not use gpt-4.1-mini and a weighted virtual model sends them there, that request is denied with 403.

%20(1).png)




















%20a%20Bad%20Idea.png)







