Istio’s networking: An in-depth look at traffic and architecture

A service mesh project like Istio introduces a number of features and benefits into your architecture, including more secure management of the traffic between your cluster’s microservices, service discovery, request routing, and reliable communication between services.

Although Istio is platform-neutral, it has become one of the more popular service meshes to use with Kubernetes. Despite this popularity, it can be complicated and difficult for someone new to service mesh to understand Istio’s networking and core mechanisms, such as:

• Envoy sidecar proxy injection
• How the sidecar intercepts and routes traffic
• Issuance of traffic management configurations
• How traffic rules take effect on the data plane

In this first post in a series of blogs explaining these mechanisms by analyzing Istio’s architecture and implementation mechanisms, we’ll cover Istio’s networking basics, the data plane and control plane, networking, and sidecar injection with Envoy Proxy. Using a demo environment, you’ll be able to see how Istio injects the init and sidecar containers along with the configuration of these containers in a pod template.

Istio’s networking basics

An overview of Istio has been covered extensively in the official documentation, but we’ll highlight the key components to review before proceeding further.

Istio consists of two main parts: the data plane and control plane.

• Data plane: The data plane, or data layer, is composed of a collection of proxy services represented as sidecar containers in each Kubernetes pod, using an extended Envoy proxy server. Those sidecars mediate and control all network communication between the microservices while also collecting and reporting useful telemetry data.
• Control plane: The control plane, or control layer, consists of a single binary called istiod that is responsible for converting high-level routing rules and traffic control behavior into Envoy-specific configurations, then propagating them to sidecars at runtime. Additionally, the control plane provides security measures enabling strong service-to-service and end-user authentication with built-in identity and credential management while enforcing security policies based on service identity.

Istio’s networking in a demo environment

Let’s create a local sandbox environment before proceeding further. This will ensure we have both an Istio service mesh deployed in Kubernetes and a sample application running in the mesh.

Tools needed:

• minikube
• istioctl (Installed with curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.11.4 sh)

Steps to deploy Istio service mesh:

1. Create a 1.22.2 version of Kubernetes cluster locally using the hyperkit driver. If you are using a non-Mac OS X machine then you will require virtualbox installed.

   minikube start --memory=4096 --cpus=2 --disk-size='20gb' --kubernetes-version=1.22.2 --driver=hyperkit -p istio-demo

2. Once the cluster is fully up, execute the following commands to set up Istio.

   # Deploy Istio operator
   istioctl operator init
   
   # Inject operator configuration
   cat << EOF | kubectl apply -f -
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   metadata:
     name: istio-control-plane
     namespace: istio-system
   spec:
     profile: minimal
     meshConfig:
       accessLogFile: /dev/stdout
       enableAutoMtls: true
       defaultConfig:
         proxyMetadata:
           # Enable basic DNS proxying
           ISTIO_META_DNS_CAPTURE: 'true'
           # Enable automatic address allocation
           ISTIO_META_DNS_AUTO_ALLOCATE: 'true'
   EOF
   

3. Deploy a sample application.

   # Create apps namespace
   kubectl create ns apps
   # Label apps namespace for sidecar auto injection
   kubectl label ns apps istio-injection=enabled
   
   # Deploy a unprivileged sleep application
   cat << EOF | kubectl apply -n apps -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: sleep
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: sleep
     labels:
       app: sleep
       service: sleep
   spec:
     ports:
     - name: http
       port: 80
     selector:
       app: sleep
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sleep
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sleep
     template:
       metadata:
         labels:
           app: sleep
       spec:
         terminationGracePeriodSeconds: 0
         serviceAccountName: sleep
         containers:
         - name: sleep
           image: curlimages/curl
           command: ["/bin/sleep", "3650d"]
           imagePullPolicy: IfNotPresent
           volumeMounts:
           - name: secret-volume
             mountPath: /etc/sleep/tls
         volumes:
         - name: secret-volume
           secret:
             secretName: sleep-secret
             optional: true
   EOF
   

4. Verify that both istio-init and istio-proxy containers are ready and running.

   kubectl get po -l app=sleep -n apps -o jsonpath='{range .items[*]}{range @.status.containerStatuses[*]}{.name},{"ready="}{.ready},{"started="}{.started}{"\n"}{end}{range @.status.initContainerStatuses[*]}{.name},{"ready="}{.ready},{"terminated="}{.state.terminated.reason}{end}' | sort

   It should show:

   istio-init,ready=true,terminated=Completed
   istio-proxy,ready=true,started=true
   

Istio sidecar containers and Envoy Proxy

Sidecar injection is one of the key functions in Istio that simplifies the process of the adding and running additional containers as part of the pod template. There are two additional containers being provisioned as part of this injection process:

• istio-init – This container configures the iptables in the application pod so that the Envoy proxy (running as a separate container) can intercept inbound and outbound traffic. Before any other container can start, Kubernetes will run it as an init container to initialize the networking in the pod. Note that allowing istio-init to manipulate the iptables in the kernel space does require escalated Kubernetes privileges. This container will automatically terminate once it has successfully completed the task. Until then, the pod will not become ready. Note that to eliminate any security complications and operational challenges when deploying this container, Istio has introduced the CNI plugin so it directly integrates with the underlying Kubernetes CNI without the need to manipulate iptables.
• istio-proxy – This is packaged as an extended version of upstream Envoy proxy. Refer to the official documentation for a list of supported extensions.

An in-depth examination of the sidecar manifest

Let’s take a look at the YAML manifest for both of these containers in the application pod we deployed earlier.

kubectl get po -l app=sleep -n apps -o yaml

We’ll look at an excerpt of both istio-init and istio-proxy containers.

istio-init container:

initContainers:
- name: istio-init
  image: docker.io/istio/proxyv2:1.11.4
  imagePullPolicy: IfNotPresent
  args:
  - istio-iptables
  - -p
  - "15001"
  - -z
  - "15006"
  - -u
  - "1337"
  - -m
  - REDIRECT
  - -i
  - '*'
  - -x
  - ""
  - -b
  - '*'
  - -d
  - 15090,15021,15020
  env:
  - name: ISTIO_META_DNS_AUTO_ALLOCATE
    value: "true"
  - name: ISTIO_META_DNS_CAPTURE
    value: "true"
  resources:
    limits:
      cpu: "2"
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      add:
      - NET_ADMIN
      - NET_RAW
      drop:
      - ALL
    privileged: false
    readOnlyRootFilesystem: false
    runAsGroup: 0
    runAsNonRoot: false
    runAsUser: 0

istio-proxy container:

containers:
- name: istio-proxy
  image: docker.io/istio/proxyv2:1.11.4
  imagePullPolicy: IfNotPresent
  args:
  - proxy
  - sidecar
  - --domain
  - $(POD_NAMESPACE).svc.cluster.local
  - --proxyLogLevel=warning
  - --proxyComponentLogLevel=misc:error
  - --log_output_level=default:info
  - --concurrency
  - "2"
  ports:
  - name: http-envoy-prom
    containerPort: 15090
    protocol: TCP
  readinessProbe:
    httpGet:
      path: /healthz/ready
      port: 15021
      scheme: HTTP
    failureThreshold: 30
    initialDelaySeconds: 1
    periodSeconds: 2
    successThreshold: 1
    timeoutSeconds: 3
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsGroup: 1337
    runAsNonRoot: true
    runAsUser: 1337
  env:
  - name: PROXY_CONFIG
    value: |
      {"proxyMetadata":{"ISTIO_META_DNS_AUTO_ALLOCATE":"true","ISTIO_META_DNS_CAPTURE":"true"}}
  - name: ISTIO_META_DNS_AUTO_ALLOCATE
    value: "true"
  - name: ISTIO_META_DNS_CAPTURE
    value: "true"
  ...

There are a few interesting things to note in these excerpts:

• Both containers are served by the same image: docker.io/istio/proxyv2:1.11. What does this mean and how does it work ? istio-iptables and proxy (under args) commands are baked into the pilot-agent binary in the image. So, if you run the pilot-agent binary in the istio-proxy container you will see this in action: kubectl exec $(kubectl get po -l app=sleep -n apps -o jsonpath="{.items[0].metadata.name}") -n apps -c istio-proxy -- pilot-agent which should result in:

  Istio Pilot agent runs in the sidecar or gateway container and bootstraps Envoy.
  
  Usage:
    pilot-agent [command]
  
  Available Commands:
    completion           generate the autocompletion script for the specified shell
    help                 Help about any command
    istio-clean-iptables Clean up iptables rules for Istio Sidecar
    istio-iptables       Set up iptables rules for Istio Sidecar
    proxy                XDS proxy agent
    request              Makes an HTTP request to the Envoy admin API
    version              Prints out build version information
    wait                 Waits until the Envoy proxy is ready
  
  

• To minimize the attack surface, securityContext stanza (which is part of the PodSecurityContext object) in the istio-init container signifies that the container runs with root privileges (runAsUser: 0), however all Linux capabilities are dropped with the exception of the NET_ADMIN and NET_RAW capabilities. These capabilities provide the istio-init init container with runtime privileges to rewrite the application pod’s iptables. This is detailed further in the Istio documentation.

      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: false
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
  

  On the other hand, istio-proxy container runs with restricted privileges as user 1337. As this is reserved, the UID (User ID) for an application workload must be different and must not conflict with 1337. The 1337 UID has been chosen arbitrarily by the Istio team to bypass traffic redirection to istio-proxy container. You can also see 1337 being used as an argument to istio-iptables when initializing iptables. As this container is actively running along with the application workload, Istio also ensures that if it’s compromised, it only has read-only access to the root filesystem.

      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1337
      runAsNonRoot: true
      runAsUser: 1337

• The istio-proxy container runs with the readiness probe shown below. Kubelet in Kubernetes uses this readiness probe to determine whether the istio-proxy is ready to accept the traffic. Kubelet will only recognize that the Pod is in the ready state if the istio-proxy container and all the corresponding application containers are in a running state and the health probes have executed successfully. If the /healthz/ready handler of the server path (defined in the pilot-agent source code) returns a successful return code, Kubelet will assume that the container is alive and healthy. failureThreshold configuration specifies the consecutive number of times this readiness probe can fail before the container is marked as unready.

    readinessProbe:
      httpGet:
        path: /healthz/ready
        port: 15021
        scheme: HTTP
      initialDelaySeconds: 1
      failureThreshold: 30
      periodSeconds: 2
      successThreshold: 1
      timeoutSeconds: 3
  

Analysis of the sidecar injection

Istio has adopted two distinct ways of injecting the sidecar proxy into the application workload: manual and automatic. Both of these methods follow the same injection principal, given “some” application workload (this can be defined as a higher level Kubernetes resource like Deployment, Statefulset, DaemonSet or even as a Pod) allows Kubernetes to inject the sidecar container using the sidecar injection template and the configuration parameters (istio-sidecar-injector configmap).

Manual sidecar injection in Istio

Out of the two methods, this is the easiest to understand. Manual injection is done via the istioctl command using the kube-inject argument. You can use either of the formats below to inject:

istioctl kube-inject -f application.yaml | kubectl apply -f -

or

kubectl apply -f <(istioctl kube-inject -f application.yaml)

When istioctl kube-inject is used to inject the sidecar, by default it will use the in-cluster configuration written as istio-sidecar-injector Kubernetes configmap. Provided are a number of flags that you can specify to customize this behavior:

--injectConfigFile string    Injection configuration filename. Cannot be used with --injectConfigMapName
--meshConfigFile string      Mesh configuration filename. Takes precedence over --meshConfigMapName if set
--meshConfigMapName string   ConfigMap name for Istio mesh configuration, key should be "mesh" (default "istio")
--injectConfigMapNam string  ConfigMap name for Istio sidecar injection, key should be "config" (default "istio-sidecar-injector")

Note that --injectConfigMapNam is a hidden flag in istioctl kube-inject that allows you to override the in-cluster sidecar injection configuration.

Alternatively, injection can be done using local copies of the configuration and the flags above:

kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.config}' > inject-config.yaml
kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.values}' > inject-values.yaml
kubectl -n istio-system get configmap istio -o=jsonpath='{.data.mesh}' > mesh-config.yaml
istioctl kube-inject \
    --injectConfigFile inject-config.yaml \
    --meshConfigFile mesh-config.yaml \
    --valuesFile inject-values.yaml \
    --filename application.yaml \
    | kubectl apply -f -

Caution must be taken to not break the sidecar when injecting it manually, especially when using custom configuration.

Automatic sidecar injection in Istio

This is considered the de facto method to inject the sidecars in Istio. This involves fewer steps to configure compared to the manual method; however, it is dependent on whether or not the underlying Kubernetes distribution has enabled support for admission controllers. Istio leverages a mutating webhook admission controller for this purpose.

Here’s the process that Kubernetes mutating admission controller handles in the sidecar injection:

1. First, the istio-sidecar-injector mutating configuration injected during the Istio installation process (shown below) sends a webhook request with all pod information to the istiod controller.
2. Next, the controller modifies the pod specification in runtime introducing an init and sidecar container agents to the actual pod specification.
3. Then, the controller returns the modified object back to the admission webhook for object validation.
4. Finally after validation, the modified pod specification is deployed with all the sidecar containers.

For the full configuration, take a look at kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml. For brevity, only two of the four webhook configurations are given in the excerpt below:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
webhooks:
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: cert
    service:
      name: istiod
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: namespace.sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: In
      values:
      - enabled
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject
      operator: NotIn
      values:
      - "false"
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: cert
    service:
      name: istiod
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: namespace.sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: In
      values:
      - enabled
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject
      operator: NotIn
      values:
      - "false"
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: cert
    service:
      name: istiod
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: object.sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: DoesNotExist
    - key: istio.io/rev
      operator: DoesNotExist
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject
      operator: In
      values:
      - "true"
    - key: istio.io/rev
      operator: DoesNotExist
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

This configuration tells the Kubernetes mutating controller to send requests to the /inject endpoint of the istiod service securely on an HTTPS port. Prior to calling the mutating webhook, Kubernetes checks to see if the user making the request is authorized to make the request. In Istio, the webhook is implemented as part of the istiod binary.

Injection can be triggered either using a label at the namespace level (istio-injection=enabled) or at the object level as an annotation (sidecar.istio.io/inject="true"). Each of the webhook configurations defines matching rules for these triggers in the namespaceSelector and the objectSelector. When the injection is based on the label defined at the namespace level, any deployment object (Deployment, StatefulSet, DaemonSet) created in the namespace will be mutated with the sidecar proxy. Below is a summary of the matching rules:

It is also possible to mutate a pod object directly (if the namespace doesn’t already have a label) when a Pod manifest is injected. Pod manifests must have a label of sidecar.istio.io/inject="true". For instance:

apiVersion: v1
kind: Pod
metadata:
  name: sleep
  namespace: apps
  labels:
    app: sleep
    sidecar.istio.io/inject: "true"
...

So far, we have looked at Istio’s networking basics, the data plane and control plane, networking, and sidecar injection with Envoy Proxy, and how Istio injects the init and sidecar containers along with the configuration of these containers in the pod template using a demo environment. In the next blog, we will analyze how iptables are configured and managed.

Reach out to us at Solo.io on Slack to find out more about Istio and our products. We also offer a number of webinars and workshops on Istio, so feel free to register to learn more.