Step by Step: Datastax Cassandra with Istio and SNI routing

Christian Posta | October 12, 2020

Cassandra is a very popular “NoSQL” database. Cassandra is a highly distributed document database that can be tolerant to certain types of failures and scaled for data-intensive microservices. As Kubernetes has become the defacto container deployment platform for microservices, running stateful workloads like Cassandra is a common choice. There are quite a few guides showing how to deploy Cassandra on Kubernetes as a StatefulSet but there are much fewer guides for connecting applications running outside of the Kubernetes cluster to the Cassandra database running inside the cluster. At Solo.io we help customers and prospects operationalize microservices networking and routing technology built on Envoy proxy like Gloo or Istio. In this blog post, we’ll dig into the details of getting a deployment of Datastax Cassandra running on Kubernetes with TLS and SNI through Istio to enable routing from outside of the cluster.

We will take a step by step approach to this guide as the components themselves are quite complex. We will build up the architecture and consume the pieces as needed explaining the approach and benefits. As we are following a specific path in this blog, and there are many considerations and tradeoffs at each step, please do reach out to us if you have questions or need help.

The architecture for this blog

There are a few ways deploy Cassandra to Kubernetes and then use Istio to control the traffic routing. In this blog post, we’re specifically considering the following architecture:

What we see in this architecture are the following salient points:

  • Cassandra deployed with the Datastax Cassandra Operator as StatefulSet
  • The Cassandra inter-node communication is secured with TLS using Cassandra’s configuration
  • Istio deployed as the service mesh treating the connections between the nodes as plaintext TCP
  • Istio ingress gateway deployed at the edge with TLS passthrough and SNI routing configured
  • Client lives outside the Kubernetes cluster with intent to connect to DB running inside cluster

In the following steps, we’ll see the following sections:

  • Deploy Istio 1.7.x
  • Deploying the Datastax Cassandra Kubernetes operator
  • Deploy a DSE CassandraDatacenter configured for TLS
  • Configure Istio ingress for TCP routing
  • Test Client
  • Configure Istio for TLS passthrough/SNI routing
  • Verify with client

Source code for this blog

You can follow along or review the source code for this blog at the following repo: https://github.com/christian-posta/dse-cass-istio-sni

Installing Istio 1.7

Go download Istio 1.7 and follow the instructions for your platform to install. For example, this blog assumes a simple installation like the following:

We will disable mTLS for Istio in this blog as we’ll be leveraging Cassandra’s built in TLS to be able to do client to server end-to-end TLS as well as SNI routing:

You can be more fine-grained about what namespaces don’t use mTLS with Istio (ie, the specific Cassandra namespace) instead of a mesh-wide setting like we used above.

Deploying Datastax Cassandra

As mentioned previously, we’ll be deploying Cassandra using the Datastax Cassandra operator. You can read the official Datastax docs for more details. We will specifically be installing the DSE operator for Kubernetes 1.16. To prepare our environment and deploy the operator run the following from the root of the source for this blog:

Let’s label the cass-operator namespace so that our Database pods get the Istio sidecar injected:

One thing to consider before deploying the operator is to disable injecting the Istio sidecar into the operator deployment. You can do that by adding the following annotation to the Kubernetes Deployment:

Let’s deploy the operator:

We are running our deployments on GKE/Google Cloud. We will provision the following StorageClass for the Cassandra Database:

Go ahead and apply it from the source code directory:

Lastly, let’s deploy the DSE CassandraDatacenter resource:

We will dig into the details of this resource in the next sections.

At this point, we should see our Cassandra nodes start to come up:

We also see the headless services that were created for this SatefuleSet:

Following the Kubernetes Service rules for StatefulSets, we can address each of the pods with the following hostnames:

  • dse-dc1-default-sts-0.dse-dc1-service.cass-operator.svc.cluster.local
  • dse-dc1-default-sts-1.dse-dc1-service.cass-operator.svc.cluster.local
  • dse-dc1-default-sts-2.dse-dc1-service.cass-operator.svc.cluster.local

However, Istio does not know about these DNS names. Istio does know about the headless services and can pull the EDS/endpoints for these, but it does not know about the specific DNS names. We can create those explicitly that look something like this for each host:

Let’s create all of the service entries:

Understanding what we’ve deployed so far

When we configured the CassandraDatacenter we configured with the following properties:

We’ve named the CassandraDatacenter resource dc1, specified a cluster size of 3 and configured resource requests and limits for the StatefuleSet that gets created. We are specifically calling out this resource section because if you don’t give Cassandra enough resources it will struggle to come up fully and appear to be stuck in a state where not all of the containers are in the Ready state.

Another part of the CassandraDatacenter config file concerns setting up TLS for the cassandra inter-node communication as well as the client TLS settings:

You can see we turn on the internode_encryption property as well as enabled: true for the client_encryption_options. These settings point to certs that live in the /etc/encryption/node-keystore.jks keystore/truststore. By default, these will be automatically created unless you create them ahead of time. So with these settings we can connect with either a plaintext or TLS client. We will try this in the next section.

Let’s take a look at the certificate/keystore as Kubernetes secrets:

You can see the dc1-ca-keystore and the dc1-keystore secrets that have been created. The first one contains the root CA that created the leaf certificates that live in the second one which is the JKS keystore.

We want to use the root CA that got created in the next sections, so let’s save that off to a file called dc1-root-ca.pem:

$ kubectl get secret -n cass-operator dc1-ca-keystore -o jsonpath="{.data['cert']}" | base64 --decode > dc1-root-ca.pem

So what we have so far if a default installation of the Datastax Cassandra using TLS between the Cassandra ring nodes with an option to use TLS from the client. The default certs/keystores were created and mounted into the StatefulSet. Let’s try routing traffic to this Cassandra cluster. After a few moments, you should see the Cassandra nodes up correctly:

Routing with Istio using TCP

We want to set up some basic TCP routing with Istio to verify everything works and was set up correctly before we start setting up the more complicated TLS/SNI routing. We will create a simple Istio Gateway and VirtualService resource that allows traffic to flow from port 9042 to the Cassandra nodes:

This Gateway resource just opens the 9042 port and expects TCP traffic. The following VirtualServiceroutes traffic from this port to the headless Cassandra service intended for routing:

Now let’s actually set this up:

Connect via simple TCP

Let’s port-forward the Istio ingress gateway so that our client can try connect on port 9042. Note, if you’re using a cloud provider or load balancer, you’ll want to verify that all firewalls are open for the 9042 port:

Now using a Cassandra client, we should be able to connect locally. For example, we can use the cqlsh client by passing some credentials (see below for getting the credentials):

Yay! We’ve connected to the DB through Istio ingress gateway using simple TCP routing. We can run a cqlshcommand to verify it works;

Connect via TLS

We saw in the previous section that we configured Cassandra to accept both plaintext and TLS client connections. Let’s try connect using cqlsh over TLS:

This should connect just like it did in the previous step.

We have configured our cqlsh rc file with the correct username/password and trusted cert.

Using Istio SNI to handle the routing

Imagine we have more than just the Cassandra database listening on the 9042 port or we need to secure and address each node individually. We can use SNI for more fine-grained routing using Istio. To do that, we need to configure Istio’s ingress gateway to use TLS Passthrough and configure our Istio routing rules to match on specific SNI hostnames. Let’s take a look.

The first thing we need to do is configure the Istio ingress gateway to treat the connections on port 9042 as TLS and use PASSTHROUGH semantics. This means the Istio ingress gateway will NOT try to terminate the TLS connection and will try to route it according to the Hostname present in the Server Name TLS extension (SNI) exchanged in the TLS ClientHello part of the handshake. We can configure the Istio ingress gateway like this:

Note we are just passing through any Hostname, though we could be very selective as required. We will match using an Istio VirtualService:

In this VirtualService we are explicitly matching on the hostnames that make up the Cassandra database. One important part to this puzzle is that under the covers the Cassandra client discovers the Cassandra nodes based on their HostID (that’s the UUID strings you see above). So to correctly set up our VirtualService we need to be able to match SNI names based on their Kubernetes DNS names as well as the internal Cassandra HostIDs. You can figure out the HostIDs and match them up to their respective Pod IP address by running the following:

You can use the Pod IP addresses listed here to match them up.

Let’s create and apply our VirtualService:

At this point, we have Istio configured to use SNI routing for the Cassandra client. In the next section, we try connect a client to this configuration.

Connecting a TLS client to Cassandra with Istio SNI routing in place

At this point, we have set up the following architecture:

Now we just need a suitable client to test this out. Unfortunately the default cqlsh CLI client does not send SNI headers, so we need to use another client. We’ve included a simple Python client with the following configuration:

Let’s source the env file (dse-client.env) and run the python client:

Conclusion

Setting up Cassandra on Kubernetes using StatefuleSets and headless services with the Datastax operator and Istio for SNI routing is very powerful but can be complex. In this blog post we saw ONE approach to doing this, however there are other options with their own tradeoffs. One part we did not cover in this blog is creating the correct certificates and SANs for each of the nodes. Please reach out to us if you need help with this kind of pattern or Istio support in general.

Special thanks to Joe Searcy and Vijay Kumar of T-Mobile for their contribution to this effort!

Back to Blog