How to configure an Istio service mesh with IPv6

Why is IPv6 becoming a standard?

Over the last decade the Internet has seen a dramatic growth which has led to the depletion of IPv4 addresses. This is attributed to the increase in adoption of smart devices such as IoT (Internet of Things) and smartphones that communicate over the internet. IPv4 has been the de facto standard until IPv6 specification was formalized and introduced in 1998.

In 2017, IPv6 was ratified by IETF (Internet Engineering Task Force) as the next generation Internal Protocol (IP) address standard intended to supplement and eventually replace IPv4.

IPv6 functions similar to IPv4 in that it provides the unique IP addresses necessary for internet devices to communicate; but it does have one significant difference: it utilizes a 128-bit IP address. In other words IPv6 supports up to 2¹²⁸ addresses (340,282,366,920,938,463,463,374,607,431,768,211,456 to be exact). Yup, that is a whole lot of IP addresses! In addition to the larger addressing space, it has a number of other key benefits over the previous standard such as better multicast routing and a simpler header format.

IPv6 implementation in Kubernetes

IPv6 was first enabled in Kubernetes (K8s) in version 1.9 as an alpha feature. The adoption of IPv6 in Kubernetes has been slow largely due to technologies like NAT (Network Access Translation) and lack of support in the underlying infrastructure with cloud providers like Google Cloud and AWS. However, over the last year there has been an increase in managed Kubernetes providers boasting IPv6 capabilities out of the box such as Platform9 and Robin.io. These products have been specifically architected to enable the rapid 5G deployments in telcos.

IPv6 has been implemented in Kubernetes in two modes:

• Standalone – single IPv6 address per pod. This is graduated to beta in release 1.18 onwards. The downside to using this mode is that it is unable to communicate with IPv4 endpoints without NAT64 translation.
• Dual stack – capable of configuring pods with both IPv4 and IPv6 address families. This support was added as an alpha feature in version 1.16 onwards and it has been graduated to beta in version 1.21.

There are a multitude of tools for bootstrapping and automating Kubernetes such as kops, kubespray, and kubeadm. As of writing this blog post, only kubeadm supports both these modes. Kops will be adding IPv6-only capability in the future as per this feature ticket.

Trying IPv6 on an Istio service mesh with Kubernetes

In this blog we will focus on running Kubernetes 1.21 in IPv6 standalone mode on AWS.

Why run Kubernetes on AWS?

All three major cloud vendors Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure support running Kubernetes natively and as managed services. However, when it comes to IPv6, AWS offers numerous IPv6 capabilities in its public cloud IaaS (Infrastructure-as-a-Service) service offerings over its competition. For instance, AWS Virtual Private Cloud (VPC) networks are IPv6 capable and the Amazon Elastic Compute Cloud (EC2) instances in those virtual networks can use DHCPv6 to obtain their IPv6 address. Currently, both GCP and Azure services lag behind AWS in terms of IPv6 offerings.

Bootstrapping Kubernetes on AWS

Before we start the provisioning process make sure to install the latest version of Terraform following these steps.

Then execute the steps below to provision the two clusters on AWS:

1. git clone https://github.com/pseudonator/terraform-bootstrap-dual-ipv6-k8-clusters
2. cd terraform-bootstrap-dual-ipv6-k8-clusters
3. terraform init
4. terraform apply

At a high level, as the following topology depicts, these steps will provision dual clusters on AWS in two independent VPCs.

Why dual clusters? As you will see later in this blog, we will be deploying and verifying IPv6 connectivity across multiple clusters using two cloud-native technologies, namely Istio and Gloo Mesh. Here are some notes on our configuration:

• Each VPC will be created with a /16 IPv4 CIDR and a /56 CIDR IPv6 block. The size of the IPv6 CIDR block is fixed and the range of IPv6 addresses is automatically allocated from Amazon’s pool of IPv6 addresses.
• In each VPC, a subnet is created with a /24 IPv4 CIDR block and a /64 IPv6 CIDR block. The size of the IPv6 CIDR block is fixed. This provides 256 private IPv4 addresses.
• Attaches an internet gateway to the VPC.
• Creates a custom route table, and associates it with the subnet, so that traffic can flow between the subnet and the internet gateway.
• Ubuntu VMs are provisioned for both master and worker Kubernetes nodes.
• A single Ubuntu VM is configured for NAT64/DNS64 translation.
• Generates a AAAA resource record for the fully qualified domain name in Route 53 for the Kubernetes API server.
• To manage strict inbound and outbound traffic between these VMs, security groups are established.

Now let’s take a closer look at the bootstrap process for Kubernetes:

• CRI (Container Runtime Interface) and cgroup driver in each Kubernetes VM (Master and Worker) are configured with containerd runtime.
• Install and set up the Kubernetes control plane with kubeadm. In this step we automatically allocate a CIDR block for pod addresses (/64) and a CIDR block for service subnets (/112) from the VPC IPv6 CIDR block of addresses.
• Deploy and configure Calico as the CNI (Container Network Interface) which provides the internal network layer.
• Registering all the worker nodes with the respective master node.
• CoreDNS configuration with DNS64 translation for returning synthetic IPv6 addresses for IPv4-only destinations.

Below diagram shows how each pod is allocated a unique IPv6 address and how the Calico CNI network overlay routes traffic between pods in the same node and across the node.

Validating IPv6 communication between the Istio clusters

To validate IPv6 traffic between both primary and secondary clusters, we will install the Istio control plane (version 1.10.3) in each of the clusters with a multi-cluster configuration followed by an installation of Gloo Mesh management plane (version 1.1.0). Gloo Mesh management plane will be deployed on the primary cluster and the relay agents will be deployed on each of the clusters. We will then add Gloo configuration to the service meshes to “stitch” the two clusters to create a single virtual mesh to route traffic between workloads. For further information, please have a look at the latest Gloo Mesh documentation to understand the core concepts.

Prerequisites:

• kubectl
• istioctl 1.10.3 (Install using curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.10.3 TARGET_ARCH=x86_64 sh -)
• meshctl 1.1.0 (Install using export GLOO_MESH_VERSION=v1.1.0 && curl -sL https://run.solo.io/meshctl/install | sh)
• Helm v3

To identify the clusters in the following installation steps, the primary cluster will be called “alpha” and the secondary cluster will be called “beta”.

We will need to copy the kubeconfig from each of the master nodes to the local machine in order to configure kubectl contexts locally:

scp -i ~/.ssh/aws_id_rsa ubuntu@<public IPv4 of alpha cluster master node>:kubeconfig ~/.kube/kubeconfig_cluster1
scp -i ~/.ssh/aws_id_rsa ubuntu@<public IPv4 of beta cluster master node>:kubeconfig ~/.kube/kubeconfig_cluster2

Setup the following environment variables to point to the appropriate kubectl context:

export KUBECONFIG=~/.kube/config:~/.kube/kubeconfig_cluster1:~/.kube/kubeconfig_cluster2
export CLUSTER1=cluster-alpha-admin@cluster-alpha && export CLUSTER2=cluster-beta-admin@cluster-beta

Verify both clusters are accessible via CLUSTER1 and CLUSTER2 environment variables:

kubectl --context=$CLUSTER1 get nodes
kubectl --context=$CLUSTER2 get nodes

Deploy Istio Control Plane

Please refer to Istio documentation which covers Istio concepts and configurations needed in a multicluster architecture.

Install the Istio Operator on both clusters:

kubectl --context ${CLUSTER1} create ns istio-operator && istioctl --context ${CLUSTER1} operator init
kubectl --context ${CLUSTER2} create ns istio-operator && istioctl --context ${CLUSTER2} operator init

Install Istio Control Plane on CLUSTER1:

kubectl --context ${CLUSTER1} create ns istio-system

cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-control-plane
  namespace: istio-system
spec:
  profile: minimal
  meshConfig:
    accessLogFile: /dev/stdout
    enableAutoMtls: true
    defaultConfig:
      envoyMetricsService:
        address: 'enterprise-agent.gloo-mesh:9977'
      envoyAccessLogService:
        address: 'enterprise-agent.gloo-mesh:9977'
      proxyMetadata:
        ISTIO_META_DNS_CAPTURE: 'true'
        ISTIO_META_DNS_AUTO_ALLOCATE: 'true'
        GLOO_MESH_CLUSTER_NAME: cluster-alpha
  components:
    cni:
      enabled: true
    ingressGateways:
      - name: istio-ingressgateway
        label:
          topology.istio.io/network: cluster-alpha-network
        enabled: true
        k8s:
          env:
            - name: ISTIO_META_ROUTER_MODE
              value: sni-dnat
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: cluster-alpha-network
          service:
            type: NodePort
            ports:
              - name: http2
                port: 80
                targetPort: 8080
              - name: https
                port: 443
                targetPort: 8443
              - name: tls
                port: 15443
                targetPort: 15443
                nodePort: 32443
    pilot:
      k8s:
        env:
          - name: PILOT_SKIP_VALIDATE_TRUST_DOMAIN
            value: 'true'
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster-alpha
      network: cluster-alpha-network
      meshNetworks:
        cluster-alpha-network:
          endpoints:
            - fromRegistry: cluster-alpha
          gateways:
            - registryServiceName: istio-ingressgateway.istio-system.svc.cluster.local
              port: 443
      istioNamespace: istio-system
      pilotCertProvider: istiod
    cni:
      excludeNamespaces:
        - istio-system
        - kube-system
      logLevel: info
EOF

Install Istio Control Plane on CLUSTER2:

kubectl --context ${CLUSTER2} create ns istio-system

cat << EOF | kubectl --context ${CLUSTER2} apply -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-control-plane
  namespace: istio-system
spec:
  profile: minimal
  meshConfig:
    accessLogFile: /dev/stdout
    enableAutoMtls: true
    defaultConfig:
      envoyMetricsService:
        address: 'enterprise-agent.gloo-mesh:9977'
      envoyAccessLogService:
        address: 'enterprise-agent.gloo-mesh:9977'
      proxyMetadata:
        ISTIO_META_DNS_CAPTURE: 'true'
        ISTIO_META_DNS_AUTO_ALLOCATE: 'true'
        GLOO_MESH_CLUSTER_NAME: cluster-beta
  components:
    cni:
      enabled: true
    ingressGateways:
      - name: istio-ingressgateway
        label:
          topology.istio.io/network: cluster-beta-network
        enabled: true
        k8s:
          env:
            - name: ISTIO_META_ROUTER_MODE
              value: sni-dnat
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: cluster-beta-network
          service:
            type: NodePort
            ports:
              - name: http2
                port: 80
                targetPort: 8080
              - name: https
                port: 443
                targetPort: 8443
              - name: tls
                port: 15443
                targetPort: 15443
                nodePort: 32443
    pilot:
      k8s:
        env:
          - name: PILOT_SKIP_VALIDATE_TRUST_DOMAIN
            value: 'true'
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster-beta
      network: cluster-beta-network
      meshNetworks:
        cluster-beta-network:
          endpoints:
            - fromRegistry: cluster-beta
          gateways:
            - registryServiceName: istio-ingressgateway.istio-system.svc.cluster.local
              port: 443
      istioNamespace: istio-system
      pilotCertProvider: istiod
    cni:
      excludeNamespaces:
        - istio-system
        - kube-system
      logLevel: info
EOF

Patch the ExternalIP of the ingress gateway services with the IPv6 address of the node where the ingress gateway pods are running. This is done because Gloo Mesh (deployed in the next section) needs to be able to discover the ingress gateway services on both clusters.

INGRESS_GW_IP=$(kubectl --context=$CLUSTER1 get nodes "`kubectl --context=$CLUSTER1 get po -n istio-system -o wide | grep istio-ingressgateway | awk {'print $7'}`" -o jsonpath='{ .status.addresses[?(@.type=="InternalIP")].address }')

kubectl --context=$CLUSTER1 -n istio-system patch svc istio-ingressgateway -p '{"spec":{"externalIPs":["'${INGRESS_GW_IP}'"]}}'

INGRESS_GW_IP=$(kubectl --context=$CLUSTER2 get nodes "`kubectl --context=$CLUSTER2 get po -n istio-system -o wide | grep istio-ingressgateway | awk {'print $7'}`" -o jsonpath='{ .status.addresses[?(@.type=="InternalIP")].address }')

kubectl --context=$CLUSTER2 -n istio-system patch svc istio-ingressgateway -p '{"spec":{"externalIPs":["'${INGRESS_GW_IP}'"]}}'

Deploy Gloo Mesh to manage Istio with IPv6

Create namespaces for Gloo Mesh management plane and the relay agents: 

kubectl --context=$CLUSTER1 create ns gloo-mesh
kubectl --context=$CLUSTER2 create ns gloo-mesh

Add the helm repository:

helm repo add gloo-mesh-enterprise https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-enterprise
helm repo update

Install the Gloo Mesh management plane on CLUSTER1 and service type to NodePort:

helm install --kube-context=$CLUSTER1 enterprise-management gloo-mesh-enterprise/gloo-mesh-enterprise --namespace gloo-mesh --version=1.1.0 --set licenseKey=$GLOO_MESH_LICENSE_KEY --set enterprise-networking.enterpriseNetworking.serviceType="NodePort"

Install the Gloo Mesh relay agent on CLUSTER1:

MGMT_INGRESS_ADDRESS=$(kubectl --context=$CLUSTER1 get svc -n gloo-mesh enterprise-networking -o jsonpath='{.spec.clusterIP}')
MGMT_INGRESS_PORT=$(kubectl --context=$CLUSTER1 -n gloo-mesh get service enterprise-networking -o jsonpath='{.spec.ports[?(@.name=="grpc")].port}')
export RELAY_ADDRESS="[${MGMT_INGRESS_ADDRESS}]:${MGMT_INGRESS_PORT}"

helm install --kube-context=$CLUSTER1 enterprise-agent enterprise-agent/enterprise-agent --namespace gloo-mesh --version=1.1.0 --set relay.serverAddress=${RELAY_ADDRESS} --set relay.cluster=cluster-alpha

Copy the root CA certificate on the alpha cluster (CLUSTER1) and create a secret in the beta cluster (CLUSTER2):

kubectl --context $CLUSTER1 -n gloo-mesh get secret relay-root-tls-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl --context $CLUSTER2 -n gloo-mesh create secret generic relay-root-tls-secret --from-file ca.crt=ca.crt

rm -f ca.crt

Similarly copy the bootstrap token:

kubectl --context $CLUSTER1 -n gloo-mesh get secret relay-identity-token-secret -o jsonpath='{.data.token}' | base64 -d > token
kubectl --context $CLUSTER2 -n gloo-mesh create secret generic relay-identity-token-secret --from-file token=token

rm -f token

Install the Gloo Mesh relay agent on CLUSTER2:

MGMT_INGRESS_ADDRESS=$(kubectl --context=$CLUSTER1 get nodes "`kubectl --context=$CLUSTER1 get po -n gloo-mesh -o wide | grep enterprise-networking | awk {'print $7'}`" -o jsonpath='{ .status.addresses[?(@.type=="InternalIP")].address }')
MGMT_INGRESS_PORT=$(k --context=$CLUSTER1 -n gloo-mesh get service enterprise-networking -o jsonpath='{.spec.ports[?(@.name=="grpc")].nodePort}')
export RELAY_ADDRESS="[${MGMT_INGRESS_ADDRESS}]:${MGMT_INGRESS_PORT}"

helm install --kube-context=$CLUSTER2 enterprise-agent enterprise-agent/enterprise-agent --namespace gloo-mesh --version=1.1.0 --set relay.serverAddress=${RELAY_ADDRESS} --set relay.cluster=cluster-beta

Register the two clusters:

cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: multicluster.solo.io/v1alpha1
kind: KubernetesCluster
metadata:
  name: cluster-alpha
  namespace: gloo-mesh
spec:
  clusterDomain: cluster.local
---
apiVersion: multicluster.solo.io/v1alpha1
kind: KubernetesCluster
metadata:
  name: cluster-beta
  namespace: gloo-mesh
spec:
  clusterDomain: cluster.local
EOF

Verify the registration with:

meshctl --kubecontext=$CLUSTER1 cluster list

Creating a virtual mesh to link the two clusters

Enable strict TLS on both clusters:

kubectl --context ${CLUSTER1} apply -f- << EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
EOF

kubectl --context ${CLUSTER2} apply -f- << EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
EOF

Inject a VirtualMesh custom resource object to create a new Virtual Mesh:

cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: networking.mesh.gloo.solo.io/v1
kind: VirtualMesh
metadata:
 name: virtual-mesh
 namespace: gloo-mesh
spec:
 mtlsConfig:
   autoRestartPods: true
   shared:
     rootCertificateAuthority:
       generated: {}
 federation:
   selectors:
   - {}
 meshes:
 - name: istiod-istio-system-cluster-alpha
   namespace: gloo-mesh
 - name: istiod-istio-system-cluster-beta
   namespace: gloo-mesh
EOF

Once injected, verify the state is ACCEPTED:

kubectl --context=$CLUSTER1 get virtualmesh -n gloo-mesh -o jsonpath='{ .items[*].status.state }'

Install applications

Create the namespace for the applications on both the clusters:

kubectl --context=$CLUSTER1 create ns apps
kubectl --context=$CLUSTER1 label namespace apps istio-injection=enabled
kubectl --context=$CLUSTER2 create ns apps
kubectl --context=$CLUSTER2 label namespace apps istio-injection=enabled

Install httpbin and sleep workloads on CLUSTER1:

kubectl --context=$CLUSTER1 -n apps apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml

cat << EOF | kubectl --context ${CLUSTER1} -n apps apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - name: httpbin
        image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        command: ["gunicorn"]
        args:
        - -b
        - '[::]:80'
        - httpbin:app
        - -k
        - gevent
        ports:
        - containerPort: 80
EOF

Install httpbin workload on CLUSTER2:

cat << EOF | kubectl --context ${CLUSTER2} -n apps apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - name: httpbin
        image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        command: ["gunicorn"]
        args:
        - -b
        - '[::]:80'
        - httpbin:app
        - -k
        - gevent
        ports:
        - containerPort: 80
EOF

Verify the workloads have been deployed on both clusters successfully. There should be two containers (application and Istio sidecar proxy) running in each workload pod.
For e.g.

and

Apply traffic policy and verify IPv6 communication between the clusters

We will inject a TrafficPolicy object to route 100% of the traffic from sleep workload on CLUSTER1 to httpbin of CLUSTER2:

cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: networking.mesh.gloo.solo.io/v1
kind: TrafficPolicy
metadata:
  name: apps-traffic-policy
  namespace: gloo-mesh
spec:
  sourceSelector:
  - kubeWorkloadMatcher:
      namespaces:
      - apps
  destinationSelector:
  - kubeServiceRefs:
      services:
        - name: httpbin
          clusterName: cluster-alpha
          namespace: apps
  policy:
    trafficShift:
      destinations:
        - kubeService:
            name: httpbin
            clusterName: cluster-alpha
            namespace: apps
          weight: 0
        - kubeService:
            name: httpbin
            clusterName: cluster-beta
            namespace: apps
          weight: 100
EOF

Verify that all the traffic flows to httpbin workload in CLUSTER2 by running the following curl command 10 times.

kubectl --context=$CLUSTER1 -n apps exec deploy/sleep -- sh -c 'for _ in `seq 1 10`; do curl -iv http://httpbin/headers; done'

You should expect to see the corresponding 10 requests reaching the Istio sidecar proxy in httpbin workload in CLUSTER2. The following command should output 10 access log lines.

kubectl --context=$CLUSTER2 -n apps logs deploy/httpbin -c istio-proxy | grep "GET /headers" | wc -l

Furthermore, at this point we can also run the official Kubernetes e2e test framework to verify the IPv6 capability against either of the clusters. Using the test tool kubetest2 provided in the e2e framework we can execute it as follows:

kubetest2 noop \
    --kubeconfig=/.kube/kubeconfig_cluster1 \
    --test=ginkgo \
    -- --focus-regex="\[Feature:Networking-IPv6\]"

If everything goes well at the end of the test it should show a result similar to,

{"msg":"Test Suite completed","total":1,"completed":1,"skipped":6674,"failed":0}

Ran 1 of 6675 Specs in 12.116 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 6674 Skipped
PASS

Ginkgo ran 1 suite in 14.113072825s
Test Suite Passed

To recap, in this blog we have looked at how to provision multiple Kubernetes clusters in IPv6 standalone mode on AWS. We then demonstrated how we can validate the communication between these two clusters using Gloo Mesh and Istio.

Reach out to us at Solo.io on Slack to find out more on how Gloo Mesh and Istio can assist with your IPv6 requirements both in single and multicluster topologies.