Challenges of running Istio distroless images

At Solo.io, we work with customers running Istio at massive scale, in secure environments, and in highly-regulated environments (FIPS/FedRAMP, PCI, etc). Our Gloo Mesh builds of Istio are based on the upstream builds with LTS (N-3) and enterprise Severity-1 response times (ie, security patching, production break/fix, feature backporting,  etc). Unlike other Istio distributions, we do not fork upstream and every feature we work on goes upstream.

When working with our customers, one thing we recommend is to run Istio with distroless builds to reduce the attack surface of the overall container and to minimize vulnerabilities in various tools that can sneak into a “distro-full” build. Our builds of Istio  (FIPS, ARM, upstream, etc) are all available with distroless images.

Challenges of running distroless

Although running distroless builds is more secure, it does introduce some challenges around usability. Some folks expect to have tools like curl, wget, or even netstat or other networking debugging tools in the images that run the Istio proxy. Others use tools like shell commands or functions like /sbin/bash sleep to work around some networking issues or app cleanup/assumption issues.

For example, we see a lot of the following in the pre-stop hook of a Kubernetes pod:

        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - sleep 5

This is typically to give some leeway to applications to finish their requests and terminate their connections gracefully. When you run distroless Istio images, however, these commands will not be available.

Even though the service mesh and its proxy should be transparent to an application, it often times proves to not be, especially when considering speciality workloads or apps not written to be cloud native.

Three challenges around the lifecycle of services in a service mesh that are made even more challenging with distroless builds of Istio include the following:

  • Waiting for the Envoy proxy to become ready before starting an app
  • Running Job workloads that run for a period of time then terminate; Envoy stays running
  • When terminating a workload, make sure that in-flight requests and any associated connections are drained gracefully

Let’s see what we can do to mitigate these challenges.

Envoy to become ready before starting the app

Having some kind of explicit ordering of startup for multiple containers in a Kubernetes Pod, especially those that run as a sidecar, has been a popular request for quite some time. However, there has not been any real solution for this yet, so until then we have to depend on some internal implementation details of Kubernetes OR build your own distroless images themselves to try and hack the app-startup problem.

The problem arises when an app’s container becomes ready before the Envoy proxy is fully ready. Since Istio’s default networking uses an init-container to redirect all app traffic to the Envoy proxy, if the proxy is not ready, it will fail. Not all apps that receive these types of failures are equipped to retry so they just fail and start a viscous crash-loop cycle.

Luckily, the Istio community has sorted this with a feature called holdApplicationUntilProxyStarts which will rely on internal Kubernetes mechanisms and knowledge about how containers get started (thanks, Marko!).

You can enable this within a distroless (or distro-full) Istio with the following config mesh-wide:


apiVersion: install.istio.io/v1alpha2
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      holdApplicationUntilProxyStarts: true

 

Or you can enable per-workload by adding the following annotation to a Pod template:


annotations:
  proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'

 

Coordinating Job lifecycle with Envoy sidecars

This usecase is a little bit tricky especially for distroless Istio builds. In this case, when a Kubernetes Job that also has the Istio sidecar injected completes successfully, the proxy continues to run and keeps the Job around until it’s cleaned up out of band. There are two viable approaches.

  1. Your app code/job should signal to the Envoy proxy to shut down
  2. You should have a “reaper” job that periodically checks Jobs and kills the pilot-agent

Although this is not very clean, it’s even more difficult in a distroless environment because you don’t have access to sleep or curl as suggested in some of the previous issues on this topic.

For option 1, signaling to the Envoy proxy to shutdown, your app would have to be aware of the proxy and send a HTTP POST to http://localhost:15020/quitquitquit. This will gracefully shutdown both pilot-agent and the Envoy proxy.

For option 2, you could have a reaper process that execs into the istio-proxy container of any Jobs that are intended to be completed and gracefully shutdownthe proxy. Luckily, the pilot-agent binary does support a way to gracefully shut down the proxy without curl. The pilot-agent inside the sidecar has a pilot-agent request command that can be used as follows:


$  pilot-agent request POST /quitquitquit

This will cleanly terminate the proxy.

Gracefully terminating connections on pod shutdown

The last usecase is about draining the requests and listeners in the proxy when an pod is given SIGTERM/SIGINT.

Envoy does have a way to drain the listeners by calling the Admin API’s /drain_listeners?graceful but luckily this is something that Istio’s pilot agent can automatically orchestrate for you. Istio has a feature called TerminationDrainDuration which allows you to insert a pause before shutting down the Envoy proxy. It will first cause the /drain_listeners?graceful endpoint, pause for the duration specified in the configuration (see below) and then call /quitquitquit on the proxy.

To configure the drain duration, you can specify it in the default Proxy configuration as follows:


apiVersion: install.istio.io/v1alpha2
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      terminationDrainDuration: 50s

Or on the workload directly with:


annotations:
  proxy.istio.io/config: '{ "terminationDrainDuration": 50s }'

This works well in Istio 1.8 and 1.9 distroless images including FIPS builds provided by Solo.io. We have also backported a way to make this work with Istio 1.7.x. Please reach out for help with this or any other Istio related issues.