As heard on Day Two DevOps
Running agents securely at scale
Thanks for listening. Solo.io's Michael Levan and Nathan Pearce joined Ned Bellavance and Kyler Middleton to unpack what changes when agents leave your laptop: sandboxing that actually constrains, state that survives, and pods that stop idling. Here's the recap, plus the fastest ways to try it yourself.
The framework from the episode
Scope, attribution, optimization
Nathan's rule for any agent, local or remote: forget the buzzwords and keep asking these three questions. Get the first two right before you chase the third.
Scope: what could it do?
Agents are very good at finding paths you didn't think of. Once one leaves your machine, the real question is blast radius: which credentials, CLIs, branches and services can it reach while you're asleep?
Attribution: what did it do?
You will miss something. When you do, you need to know which agent took which action, and how many tokens it spent on that specific task. That's a problem unique to agentic workloads.
Optimization: how do we do better?
This stack is new and maturing fast. Keep the first two in place and you can iterate the DevOps way: measure, improve, repeat.
Key takeaways
Six things to remember from the conversation
Each takeaway maps to a question listeners are asking as they move agents from the IDE into production.
01 · State
Agents brought state back
We spent a decade making workloads stateless so pods could be nuked and recreated. Agents need their session and context to survive, which breaks Lambda-style and plain pod patterns.
02 · Sandboxing
Deny all, then open as needed
A container with full ingress and egress isn't much of a sandbox. Start with zero access, then route each LLM or MCP server through agentgateway as the agent proves it needs it.
03 · Isolation
Know which isolation you're getting
Software isolation such as gVisor runs in user space. Hardware isolation with microVMs gives each agent its own guest kernel. Ask vendors what "sandbox" really means underneath.
04 · Idle cost
Scale to zero, wake up warm
Agents sit idle most of the time, like an unanswered Slack thread. Agent Substrate suspends them to a warm state that holds no resources but keeps the session, so they come back fast.
05 · Snapshots
Snapshot, restore and hand off
Sessions are snapshotted to storage you choose, such as S3 or Google Cloud Storage, so you or a teammate can pick up tomorrow. Keep images lean: a giant npm install slows restores.
06 · Experience
Nobody should have to curl an agent
kagent sits on top of Substrate with CRDs, a UI and a harness for Claude, Codex or your own framework, so platform teams and HR, finance or marketing get governed, observable agents.
How the stack fits together
Like Kubernetes, many parts working as one
Michael compared Agent Substrate to a Kubernetes cluster: many components you treat as one platform, organized around one mission: preserve the agent's state.
kagent: the experience layer
Declarative agents as Kubernetes resources, a UI and terminal access, token usage and full traces, plus access policies so each team only sees its own agents.
Agent Substrate: the runtime
Workers and worker pools run your actors (agents). Kubernetes handles scheduling; the Substrate API handles fast spin-up, suspend and snapshot/restore. Open source, with weekly community calls.
agentgateway: the front door
Ingress and egress for every agent, with L7 policy, rate limiting, prompt guarding, OBO token exchange and audit for traffic to LLMs and MCP servers.
Pick your next step
Everything discussed on the show is open source and free to try. Start hands-on, go deeper with the team, or meet the community in person.
Listener questions
Agent questions, answered
The questions Ned and Kyler asked on behalf of practitioners, with the short answers.
Is Kubernetes the only way to run agents at scale?
Not the only way, but Michael's view on the show is that it's still the best place to orchestrate and schedule workloads, and most enterprises already run it. Any framework, including CrewAI or LangGraph, can run there too.
What makes a sandbox different from a container?
A container is isolated, but it can still have full ingress, egress and east-west access. A real agent sandbox starts with no access and adds it deliberately, and you should know whether isolation is software-level (such as gVisor) or hardware-level (microVMs).
Where is an agent's state kept?
Agent Substrate snapshots sessions to the object storage you choose, such as S3 or Google Cloud Storage, and restores them, routing you back to the right session when you or a teammate return.
Can I use the AI tools I already pay for?
Yes. kagent's harness object lets you run agents with Claude, Codex, the kagent harness or your own framework, using the provider subscriptions you already have.
Is it open source? What does enterprise add?
kagent, Agent Substrate and agentgateway are open source, and Solo.io engineers are among the Substrate maintainers. Enterprise adds full observability, OBO token exchange, governance, policy enforcement and support.
How do I know if I need this?
If you're taking agents to production and want control over what they can do and what they did, you're a candidate. As Nathan put it, anything that only runs when it's doing work for you is a strong security pattern.
Ready to take agents to production?
Talk with Michael, Nathan and the team about sandboxing, scale-to-zero and governance for your agents, or get hands-on first in a free lab.
