Prevent security risks with multi-cluster Istio deployments
Here at Solo.io we treat security very seriously. In order to help ensure good security, when we’re aware of a potential weakness, we aim to address it as quickly and completely as possible. Here’s an example related to istiod, kubeconfig, and multi-cluster Istio service meshes.
Istio had a potential CVE report related to the use of untrusted kubeconfig files which can unintentionally give others access to run malicious code on a Kubernetes cluster. This vulnerability was first reported in August (see issue here.) This impacts the security of service meshes during Istio multi-cluster setup when istiod has access to a remote cluster’s kubeconfig files.
As the author notes –
“There are two broad set of issues:
- A Kubeconfig file exposes users (Kubernetes administrators, developers) to the risk of their local systems being compromised.
- Automated systems like public CI/CDs or other similar services (dashboards, service meshes, etc.) may get compromised through the sdk client executing insecure operations.”
The second bullet makes clear that this is something that Istio service meshes will need to address. We’ve covered this topic in previous blogs and sessions, which have also mentioned related security issues with basic open source Istio and some vendors’ offerings. Let’s take a look…
In their talk at ServiceMeshCon way back in May 2021, Administering Multi Cluster Service Meshes Securely, Eric Murphy and Eitan Yarmush outlined the risks like this:
In her VMblog in September 2021, Multi-cluster Service Mesh is Still Hard, Lin Sun notes that: “You don’t always want to have all of your services exposed out of the cluster and make them available to other clusters by default. The best security practice is to allow nothing by default and gradually enable access control as required.”
Using multi-cluster and Gloo Mesh for security risk prevention
Unfortunately, not everyone is taking this concern seriously. As we wrote in this blog in November 2021, Gloo Mesh vs. other Istio products – what we’ve learned over the past year: “Google’s Anthos Service Mesh (ASM) even recommends setting up Istio community’s endpoint discovery service for multi-cluster, which we’ve found with our customers to be insecure. With this model, you basically create permissions in each Kubernetes cluster to communicate with every other clusters’ Kubernetes API directly. If one cluster gets compromised, all clusters are potentially compromised.”
We believe that giving istiod this kind of access for convenience is the wrong approach. Our enterprise users have shared with us security concerns with this approach because they don’t want the Kubernetes API server access credentials to leave the cluster which could be maliciously used for other purposes.
A better approach is used in Gloo Mesh’s multi-cluster setup which is inherently more secure and protects our customers from being exposed to this exploit. Gloo Mesh customers are simply not exposed to this vulnerability as we don’t give istiod any access to other clusters. Instead Gloo Mesh follows a federation model and uses a relay-communication architecture instead of direct Kube API access. For the organizations with which we work, this provides a much more secure posture.
Learn more about multi-cluster service mesh and security
Want to learn more about how we can help you keep your environment secure? Check out these resources!
