Operating, Isolating, and Scaling Agent Workloads on Kubernetes w/ Agent Substrate

One agent is a deployment. A platform full of them is a different problem. Run as ordinary pods, every agent holds its memory while it waits for the next message, carries a provider key it can read, calls the provider directly, and shares the node's kernel with every other team's agents.

This solution shows you how to run agents on Solo Enterprise for kagent and Agent Substrate, so that each session is its own sandbox that holds no memory between turns, no agent ever holds a key, and every model and tool call goes through Solo Enterprise for agentgateway.

‍

Eight agents either way: eight pods, or two workers and an object store.

‍

‍

Common challenges
‍

• Idle agents that still hold a pod

An agent waits for a person, thinks for thirty seconds, calls a few tools, and goes quiet again. Run as a pod per session it holds its memory the whole time, whether or not anything is happening inside it.

‍

• The key in the agent's environment

A provider key in an environment variable is readable by anything that can read the process, including the code the model asks the agent to run. Every agent that holds one is a place for it to leak from.

‍

• One kernel, every team

Pods on a node share its kernel. An agent that runs code it was handed by a model is one container escape away from every other team's agents on that node.

‍

• No place to stand

Each agent calls its provider directly, so there is no single place to authenticate agents, stop a runaway job, decide where an agent may send data, or say which team spent what.

‍

‍

‍

What you build
Diagram of Kubernetes agent deploying actors in worker pods, interacting with LLM providers, MCP servers, and other agents.
Solution Architecture – Scaling Agent Workloads on Kubernetes (Coming Soon)

‍

‍

  • One sandbox per session. Each session is an Agent Substrate actor in its own gVisor sandbox. It is restored from a snapshot when a message arrives and suspended when the turn ends.
    ‍
  • Memory follows turns, not sessions. A suspended session holds no memory, only a snapshot of its volumes. Workers are sized for the turns in flight at your busiest moment, not for every session open.
    ‍
  • No agent holds a key. The agent is given a placeholder. Substrate's egress gateway puts the real credential on the request after it leaves the sandbox, and the provider key itself is stored once, in the gateway.
    ‍
  • One way out. Every model and tool call goes through Solo Enterprise for agentgateway, and each agent may reach only the destinations its egress policy allows.
    ‍
  • Budgets and attribution per team. Daily token budgets enforced at the gateway, before the provider is called, and every token attributed to the team and key that spent it.
    ‍
  • Your identity provider, your groups. The UI and API sign in against your identity provider, and access policies keyed to its groups decide who can see, change and talk to each agent. A session belongs to the user who started it.
    ‍
  • A trace for every turn. Each turn is traced from the controller to the provider and kept in your cluster, with the session, the user and the tokens on it, and access to traces governed like everything else.

‍

‍

What Solo Enterprise for kagent adds

The open source stack runs the same architecture: actors, sandboxes, a gateway in front. What it leaves to you is everything a platform team is asked about once more than one team depends on it. Solo Enterprise for kagent and Solo Enterprise for agentgateway close those gaps.

‍

Capability Open source Solo Enterprise
Run and scale
Sandboxed execution by default, with gVisor and microVM support✓✓
Idle sessions suspended to snapshots and restored on demand✓✓
High-density shared worker pools, autoscaled through Kubernetes HPA✓✓
Claude Code and Codex harnesses alongside declarative agents✓✓
Checkpoints, session forking and scheduled runs✓✓
Isolate
Credentials injected at egress: no provider key in the agent✓✓
Inbound and outbound agent traffic through agentgateway, under policy✓✓
Actor identity, with mTLS on ingress and egress✓✓
Govern
Single sign-on through OIDC (Keycloak, Entra ID, Okta), verified by kagent✗✓
Catalog access control by user and group✗✓
Daily token budgets per team✗✓
Operate
Agent trace integration✓✓
Telemetry storage, analytics and health dashboards✗✓
Per-agent analytics: wake-up latency, cold starts, request volume✗✓
Live view of Substrate and agent lifecycle, with checkpoint browsing✗✓
Deploy and support
Runs on older Kubernetes, without the beta certificate APIs✗✓
Specialty builds for security patches and compliance, including FIPS✗✓
Enterprise production support✗✓

‍

‍

What you can answer afterwards

  • What does an idle agent cost us while nobody is talking to it?
  • Can an agent read its own provider key?
  • Where is each agent allowed to send data?
  • Which team spent what, and what stops a runaway job?
  • Who can see, change and talk to this agent, and read its traces?
  • How many workers do we need for our busiest hour?

‍

‍

Common questions

Q. How do I run many agents on Kubernetes without a pod per agent?

Run them as actors on a shared worker pool. With Agent Substrate each session is restored onto a warm worker when a message arrives and suspended to an object store when the turn ends, so only sessions that are working occupy memory. A suspended session holds no memory, only a snapshot of its volumes.

‍

Q. Does Agent Substrate use less memory than a pod per agent?

No, and the guide says so. A pod per agent uses less memory, but it shares the node's kernel and keeps every session of that agent in one process. Against a pod per session, which isolates sessions the way Substrate does, Substrate holds memory only for the sessions that are working.

‍

Q. How do I keep provider API keys out of agent containers?

Give the agent a placeholder and inject the credential outside it. kagent hands the agent a placeholder value, Substrate's egress gateway swaps it for the agent's gateway key on the way out, and the gateway holds the provider key. In the validated deployment the agent's environment held only the placeholder, and a search of its sandbox and of its suspended snapshot found neither key.

‍

Q. How do I size a worker pool for agents?

Size it for turns in flight. A worker runs as many turns at once as the CPUs it advertises, and each running actor reserves 1 CPU and 1 GiB until it suspends. A turn over that limit waits for a slot instead of failing.

‍

Q. Why not run the open source stack in production?

You can run it, and the open source guide shows how. What you take on is the governance around it: verifying who is signed in, deciding who may use each agent, budgeting spend per team, keeping traces under access control, and running on clusters without Kubernetes' beta certificate APIs. Solo Enterprise ships those, and Solo supports them.

‍

Q. Can I run Claude Code as an agent runtime?

Yes. A Harness chooses the runtime for each agent, and the guide runs the same agent on kagent's own runtime and on Claude Code side by side. Budget for the difference: a Claude Code turn is charged many times the tokens of a kagent turn, almost all of it Claude Code's own system prompt.

‍

‍

‍

‍

‍