Running Istio Ambient Mesh With Any CNI
In a recent Solo.io live stream, I spoke with Yuval Kohavi, Solo.io’s chief architect, about an exciting feature called in-pod redirection mode that’s going to be available in the upcoming Istio release.
The in-pod redirection mode re-thinks the way traffic is intercepted in the Istio Ambient Mesh, enabling it to work together with any CNI.
Current Implementation
Istio Ambient mesh relies on a component called istio-cni that runs on each node in the cluster. As pods are added or removed from the ambient mesh, the istio-cni configures traffic redirection on the node in such a way that any inbound and outbound traffic gets intercepted and redirected to the ztunnel component running on the same node.
The diagram above shows a detailed overview of the traffic flow when a request is sent from a sleep pod to httpbin endpoint and how it’s captured on the node. This involves checking if the pod is part of the ambient mesh (by checking the IPSet), then the packets flowing through the iptables rules and marking them so they get routed to the correct interface (istioout) on the node, which is connected to an interface (pistioout) in the ztunnel pod.
One of the issues of doing the interception and redirection in the node’s network namespace is that Istio Ambient Mesh is not the only player here. Cluster’s primary CNI uses the same network namespace to set up cluster networking. The two configurations could easily interfere with each other, causing conflicts.
There are multiple CNI implementations out there and coming up with a universal approach and implementation that would work with all of them is almost impossible.
The Novel In-Pod Implementation
Since host network namespace was out of the picture – there’s no option of configuring redirection there – the focus went back to the sidecar mode to figure out if there’s a way to mimic what sidecars are doing and configure the redirection inside the pod’s network namespace.
The key to the in-pod implementation was discovering that a Linux process running in one network namespace can create listening sockets within another network namespace, allowing us to configure ztunnel to set up listening sockets within each pod’s network namespace.
Thus, the in-pod redirection was born! With this approach, ztunnels on the node can configure redirection sockets inside each pod’s network namespace.
The top portion of the diagram shows the outgoing traffic flow. When a request is sent from the pod, the iptables within the pods network namespace capture the traffic and route it to the 15001 port, which the ztunnel pod listens on. The traffic is processed through the ztunnel and then sent from the pod’s network namespace to the destination.
Similarly, on the inbound the traffic is captured by the iptables inside the pod, redirected to the port 15008, processed through the ztunnel, and then sent to the application running inside the pod.
The istio-cni component still plays a role in this solution – it detects the pods that are added to the ambient mesh, configures the redirection inside the pod’s network namespace, and notifies the ztunnel and tells it to configure the listening ports.
The best thing about this solution is that the node, application running in the pod and the CNIs are completely unaware of this redirection. Everything looks to them like the traditional sidecar model, without any sidecars!
Implementing This Feature
This solution means any Istio policies as well as the CNI’s NetworkPolicy resource can be applied and enforced at the same time. You can check out the demo of how this works in the Ambient Mesh with CNI video.
For a more detailed and in-depth look into the implementation, head to the Maturing Istio Ambient: Compatibility Across Various Kubernetes Providers and CNIs blog post.
This feature will be available in Istio’s upcoming 1.21 release. While Istio Ambient Mesh is currently progressing from alpha toward beta, our team is ready to support as you explore its capabilities, and as the technology develops. Get started with Ambient Mesh here.
