Navigating Istio Config: a look into Istio’s toolkit
Let’s talk about debugging Istio by looking directly at the generated Envoy configuration…. An Istio service mesh gives you dynamic traffic control capabilities by deploying an Envoy Proxy next to each of your containers and then programming it with a configuration to drive your desired behavior. Typically, this configuration can be hundreds or thousands of lines even in modestly sized environments. Most users will never have to actually look at this configuration, but when things are not working as expected, reading your configuration can be a vital debugging tool.
You can look at the entire Envoy configuration for any given deployment by running
istioctl dashboard envoy deploy/<name>and going to http://localhost:15000/config_dump . Even in a small environment with just a few workloads, we have over 16,000 lines of YAML, which makes it very difficult to find what we want to see. Fortunately, Istio’s
istioctl CLI tool also comes with a
proxy-config sub command that makes reading this a little bit easier by allowing you to see portions of the file. Istio.io has documentation on how to use this command for debugging. However, this content is a bit low level and can be too advanced for users who are not familiar with Envoy. In this blog, we use screenshots and pointers to highlight the most important bits of an Envoy configuration that you can use when you are debugging.
Let’s use the sleep and httpbin samples as an example to see this in action. Deploy the httpbin and sleep applications to an Istio enabled namespace:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml kubectl scale deploy/httpbin --replicas=2 kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml
Your pods should look something like this:
Now, we’ll look at the relevant config that comes into play when sleep talks to one of the two httpbin pods:
kubectl exec deploy/sleep -c sleep -- curl http://httpbin:8000
The Envoy sidecar container intercepts this curl http request from the sleep container and decides where and how to send the request based on its configuration. At a basic high level, this client side Envoy configuration can be broken down to listeners, routes, clusters and endpoint. A listener tells Envoy to bind to a specific port and uses filters telling it what to do, such as connect to a route configuration. A route has a list of domains that if matched, maps the request to an Envoy cluster. A cluster is then made up on a set of IP addresses which are the endpoints that back that cluster.
Let’s use istioctl’s
proxy-config , or
pc for short to look at each one of these in more detail.
The above screenshot should be helpful in quickly determining where the routing, retry, circuit breaking, locality and failover configuration is set in Envoy. This is not only useful in determining if it’s configured to match what you set with Istio, but to also see what the default settings are.
At Solo, we have customers who are deploying some of the largest deployments of service mesh and Istio in the world. We have an amazing team of field engineers who are experts in Envoy and Istio, and work closely with our customers every day.
If you have any questions, reach out to our experts!