LiteLLM alternative
The Secure, High-Performance Alternative to LiteLLM
Your AI gateway holds the keys to every model and tool your agents use, which makes it a target. agentgateway is built in Rust with layered defenses for LLM, MCP, and agent-to-agent traffic, adds a fraction of a millisecond to every request, and runs as a standalone binary or natively on Kubernetes. Hosted by the Linux Foundation and backed by Solo.io.
Performance
Benchmarked on LiteLLM’s own test suite
John Howard, an agentgateway maintainer at Solo.io, ran agentgateway through the open benchmark BerriAI built to compare AI gateways. Same harness, same mock, same machine.
p99 latency added per request
peak memory, lowest of five gateways
p99 at 30K requests/sec in Gateway API tests
p99 added latency
Milliseconds the gateway adds, log scale. Lower is better.
Peak memory
Resident memory under the same load. Lower is better.
Results from John Howard’s run of BerriAI’s open AIGatewayBench with agentgateway added (pull request #2, under review). Same host and local deterministic mock for every gateway, Anthropic Messages API, n=5,000 per endpoint. Overhead is gateway p99 minus direct-to-mock p99. Measures gateway overhead, not model inference. Gateway API figure from Designing agentgateway (AAIF).
Security
Defense in depth for every AI request
An AI gateway stores credentials for every model provider and tool your company uses, so attackers go after it directly. agentgateway layers identity, policy, guardrails, and credential handling into the data plane itself.
| Threat | How agentgateway defends against it |
|---|---|
| Poisoned package releases | Ships as a compiled Rust binary in a container image, so there is no install-time script or interpreter startup hook for injected code to ride on. Solo Enterprise for agentgateway adds LTS releases, CVE patches, and a published SBOM. |
| Stolen provider keys | Credentials are injected at the gateway on the way out, so agents and apps never hold them. Token exchange (RFC 8693, JWT assertion, Entra on-behalf-of) replaces long-lived keys where providers support it. |
| Injection through admin or setup paths | Routes, backends, policies, and MCP servers are declared up front, in a config file in standalone mode or in Gateway API resources governed by Kubernetes RBAC, rather than created through the gateway’s request path. The admin interface listens on loopback only by default, and the UI can sit behind OIDC. |
| Authentication bypass | JWT, OAuth, API keys, external auth, and OPA are enforced on every route, with CEL-based RBAC down to the model and MCP tool. External auth fails closed. |
| Data leaking through prompts and tools | Prompt guards and moderation (Bedrock Guardrails, Google Model Armor, OpenAI moderation), WAF, and, in Solo Enterprise, Microsoft Purview DLP. |
| Memory-safety bugs | Written in Rust, which rules out whole classes of buffer overflow and use-after-free vulnerabilities common in C and C++ proxies. |
2026 incidents across the AI gateway ecosystem included compromised package releases, pre-auth SQL injection, and command injection through MCP setup endpoints. See LiteLLM’s October 21 incident report and The Hacker News.
Run it your way
Standalone or on Kubernetes, your choice
agentgateway no longer requires Kubernetes. Run it as a standalone binary or as a Kubernetes-native gateway, with the same Rust data plane either way.
Standalone
A single binary, anywhere
No Kubernetes required. Run agentgateway on bare metal, a VM, or a laptop, or as a Docker container.
- ●One config file you can also edit from the built-in UI
- ●Built-in admin UI with cost and analytics dashboards
- ●Admin interface on loopback only by default
- ●Ideal for developers, small teams, and edge deployments
Kubernetes
Native to your platform
A control plane that implements the Kubernetes Gateway API, with full conformance.
- ●Configured with Gateway API and agentgateway custom resources
- ●Fits GitOps workflows and Kubernetes RBAC
- ●Installed with Helm, scaled like any other workload
- ●Ideal for platform teams running shared infrastructure
One gateway for every protocol
Route and secure LLM, MCP, agent-to-agent (A2A), REST, and gRPC traffic with the same policies, instead of running a separate proxy for each.
Same data plane in both modes
Standalone and Kubernetes deployments run the same Rust proxy with the same security features, so you can start small and move to Kubernetes without switching gateways.
Open governance
An Apache 2.0 project of the Agentic AI Foundation under the Linux Foundation, with 300+ contributors from 60+ organizations.
Moving from LiteLLM
Switch without a rewrite
LiteLLM made multi-provider routing easy, and many teams start there. When it’s time for production-grade security and performance, the move is incremental.
Step 1
Run side by side
Deploy agentgateway next to your existing gateway. agentgateway speaks the OpenAI and Anthropic APIs, so most clients only need a new base URL.
Step 2
Move credentials to the gateway
Shift provider keys into gateway-managed credentials, then turn on token exchange and credential injection where you can.
Step 3
Switch route by route
Move traffic one model or team at a time, and add guardrails, rate limits, and MCP policies as you go.
Get hands-on
See agentgateway in action
Self-paced labs from the engineers who build agentgateway.
FAQ
Frequently asked questions
What is agentgateway?
agentgateway is an open source, AI-native gateway written in Rust. It secures and governs traffic to LLMs, MCP tools, and other agents, alongside REST and gRPC, under one set of policies. It is a project of the Agentic AI Foundation under the Linux Foundation, and Solo.io offers Solo Enterprise for agentgateway with 24x7 support.
How is agentgateway different from LiteLLM?
LiteLLM is a Python SDK and proxy that made multi-provider LLM routing easy. agentgateway is a gateway written in Rust that runs standalone or natively on Kubernetes and handles LLM, MCP, agent-to-agent, REST, and gRPC traffic with layered security built into the data plane. In John Howard’s run of BerriAI’s own AIGatewayBench, agentgateway added 0.02 ms of p99 latency and used 40.6 MB of peak memory, the lowest of the five gateways tested.
How does agentgateway protect provider API keys?
agentgateway injects credentials into outbound requests at the gateway, so agents and applications never hold them. Where providers support it, token exchange (RFC 8693, JWT assertion, and Microsoft Entra on-behalf-of) replaces long-lived keys with short-lived tokens.
How does agentgateway reduce supply chain risk?
agentgateway ships as a compiled Rust binary in a container image, with no package install step or interpreter startup hooks for injected code to run through. Solo Enterprise for agentgateway adds long-term-support releases, CVE patches, and a published software bill of materials.
Do I need Kubernetes to run agentgateway?
No. agentgateway runs in two modes. Standalone mode is a single binary or Docker container configured with one file, with a built-in admin UI and cost dashboards. Kubernetes mode adds a control plane that implements the Kubernetes Gateway API for teams that manage infrastructure with GitOps. Both modes use the same Rust data plane.
Is agentgateway also an MCP gateway?
Yes. It federates MCP tool servers behind one endpoint, implements the MCP authorization spec with OAuth, and applies tool-level access policies, so you control which agents can call which tools.
Do I need an enterprise contract to use agentgateway?
No. agentgateway is open source under Apache 2.0 and free to run on your own. Solo Enterprise for agentgateway adds enterprise features such as advanced guardrails and DLP integrations, LTS releases, and 24x7 support.
Put a secure gateway in front of your AI traffic
Request a demo to see how agentgateway secures LLM, MCP, and agent-to-agent traffic in your environment, and get a plan for moving from LiteLLM.
