Multi-Cluster Service Mesh Role-Based-Access-Control

Denis Jannot | November 12, 2020

In the previous posts, we covered how Gloo Mesh (previous known as Service Mesh Hub) makes it easy to federate the identity across clusters to allow cross-cluster communication, failover and access control based on policies.

RBAC in Kubernetes

In large organizations, several teams are using the same Kubernetes cluster. They use Kubernetes RBAC to define who can do what and where.

One common approach is to create one or several namespaces for each team and to define a Kubernetes namespace admin Role as follow:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
  metadata:
  namespace: default
name: namespace-admin
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["*"] # any resource
  verbs: ["*"] # any action

Then, you would assign this role to a user using a Kubernetes RoleBinding:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: default-namespace-admin
  namespace: default
subjects:
- kind: User
  name: user1
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: namespace-admin
  apiGroup: rbac.authorization.k8s.io

RBAC in single cluster Service Mesh

Istio Authorization Policy enables access control on workloads in the mesh.

In a single cluster Service Mesh deployment, Kubernetes Roles and RoleBindings can be used to determine which users are allowed to create these Istio AuthorizationPolicies.

For example, the Kubernetes namespace admin Role we created above could be updated to allow a namespace admin to create Istio AuthorizationPolicies:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
  metadata:
  namespace: default
name: namespace-admin
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["*"] # any resource
  verbs: ["*"] # any action
- apiGroups: ["security.istio.io/v1beta1"]
  resources: ["AuthorizationPolicy"] # any resource
  verbs: ["*"] # any action

This approach can be followed for all the other Istio Custom Resource Definitions (CRD).

RBAC in multi cluster Service Mesh

Gloo Mesh provides Custom Resource Definitions to define policies globally (across multiple clusters).

A TrafficPolicy applies between a set of sources (workloads) and destinations (traffic targets), and is used to describe rules like for every request to services on cluster C, increase the timeout and add retries. Traffic policies support timeouts, retries, CORS, traffic shifting, header manipulation, fault injection, subset routing, weighted destinations, and more.

An AccessPolicy also applies between sources (this time representing identities) and destinations, and is used to finely control which services are allowed to communicate together. On the virtual mesh, a user can specify a global policy to restrict access, and require users to specify access policies in order to enable communication to services.

Gloo Mesh translate the TrafficPolicies and AccessPolicies into Istio resources on the local clusters.

So, how do we manage RBAC now ? How can we define who is allowed to create these Gloo Mesh policies ? We can’t use Kubernetes Roles because we need to be more fine grained.

We need to be able to:

define which cluster a user can route traffic to
define which namespace a user can create policies for
define which type of policies a user can create

That’s why we have implemented a very powerful RBAC mechanism in Gloo Mesh.

Gloo Mesh RBAC in Action

Let’s take the common example of a namespace admin again and extend this concept across clusters.

In the previous post called Cross-cluster service communication with service mesh, we wanted to allow the productpage service on one cluster to communicate with the reviews service running on another cluster:

To make it possible, we created the following TrafficPolicy:

apiVersion: networking.smh.solo.io/v1alpha2
kind: TrafficPolicy
metadata:
  namespace: service-mesh-hub
  name: simple
spec:
  destinationSelector:
  - kubeServiceRefs:
      services:
        - clusterName: cluster1
          name: reviews
          namespace: default
  trafficShift:
    destinations:
      - kubeService:
          clusterName: cluster2
          name: reviews
          namespace: default
          subset:
            version: v3
        weight: 75
      - kubeService:
          clusterName: cluster1
          name: reviews
          namespace: default
          subset:
            version: v1
        weight: 15
      - kubeService:
          clusterName: cluster1
          name: reviews
          namespace: default
          subset:
            version: v2
        weight: 10

If we enable Gloo Mesh RBAC and we try to create this policy, we get the following output:

Error from server (User kubernetes-admin does not have the permissions necessary to perform this action.): error when creating “STDIN”: admission webhook “rbac-webhook.service-mesh-hub.svc” denied the request: User kubernetes-admin does not have the permissions necessary to perform this action.

Gloo Mesh comes with a pre configured admin-role:

apiVersion: rbac.smh.solo.io/v1alpha1
kind: Role
metadata:
  name: admin-role
  namespace: service-mesh-hub
spec:
  trafficPolicyScopes:
    - trafficPolicyActions:
        - ALL
      trafficTargetSelectors:
        - kubeServiceMatcher:
            labels:
              "*": "*"
            namespaces:
              - "*"
            clusters:
              - "*"
        - kubeServiceRefs:
            services:
              - name: "*"
                namespace: "*"
                clusterName: "*"
      workloadSelectors:
        - labels:
            "*": "*"
          namespaces:
            - "*"
          clusters:
            - "*"
  virtualMeshScopes:
    - virtualMeshActions:
        - ALL
      meshRefs:
        - name: "*"
          namespace: "*"
  accessPolicyScopes:
    - identitySelectors:
        - kubeIdentityMatcher:
            namespaces:
              - "*"
            clusters:
              - "*"
          kubeServiceAccountRefs:
            serviceAccounts:
              - name: "*"
                namespace: "*"
                clusterName: "*"
      trafficTargetSelectors:
        - kubeServiceMatcher:
            labels:
              "*": "*"
            namespaces:
              - "*"
            clusters:
              - "*"
          kubeServiceRefs:
            services:
              - name: "*"
                namespace: "*"
                clusterName: "*"
  failoverServiceScopes:
    - meshRefs:
        - name: "*"
          namespace: "*"
      backingServices:
        - kubeService:
            name: "*"
            namespace: "*"
            clusterName: "*"

As you can see, a Gloo Mesh allows us to be very fined grained about the permissions we grant.

Let’s create a Gloo Mesh RoleBinding to grant the current user this admin role:

apiVersion: rbac.smh.solo.io/v1alpha1
kind: RoleBinding
metadata:
  labels:
    app: service-mesh-hub
  name: admin-role-binding
  namespace: service-mesh-hub
spec:
  roleRef:
    name: admin-role
    namespace: service-mesh-hub
  subjects:
    - kind: User
      name: kubernetes-admin

Now, if we try to create the TrafficPolicy again, it works:

trafficpolicy.networking.smh.solo.io/simple created

As we stated at the beginning of the post, it’s very common to allocate one or several namespaces to each team.

So, let’s create a global namespace admin role:

apiVersion: rbac.smh.solo.io/v1alpha1
kind: Role
metadata:
  name: default-namespace-admin-role
  namespace: service-mesh-hub
spec:
  trafficPolicyScopes:
    - trafficPolicyActions:
        - ALL
      trafficTargetSelectors:
        - kubeServiceMatcher:
            labels:
              "*": "*"
            namespaces:
              - "default"
            clusters:
              - "*"
        - kubeServiceRefs:
            services:
              - name: "*"
                namespace: "default"
                clusterName: "*"
      workloadSelectors:
        - labels:
            "*": "*"
          namespaces:
            - "default"
          clusters:
            - "*"
  virtualMeshScopes:
    - virtualMeshActions:
        - ALL
      meshRefs:
        - name: "*"
          namespace: "default"
  accessPolicyScopes:
    - identitySelectors:
        - kubeIdentityMatcher:
            namespaces:
              - "default"
            clusters:
              - "*"
          kubeServiceAccountRefs:
            serviceAccounts:
              - name: "*"
                namespace: "default"
                clusterName: "*"
      trafficTargetSelectors:
        - kubeServiceMatcher:
            labels:
              "*": "*"
            namespaces:
              - "default"
            clusters:
              - "*"
          kubeServiceRefs:
            services:
              - name: "*"
                namespace: "default"
                clusterName: "*"
  failoverServiceScopes:
    - meshRefs:
        - name: "*"
          namespace: "default"
      backingServices:
        - kubeService:
            name: "*"
            namespace: "default"
            clusterName: "*"

If we update the `RoleBinding` to assign this new `Role` instead of the previous one, we can still create the previous TrafficPolicy, but we can’t create the following one that target another namespace:

apiVersion: networking.smh.solo.io/v1alpha2
kind: TrafficPolicy
metadata:
  namespace: service-mesh-hub
  name: simple-ns1
spec:
  destinationSelector:
  - kubeServiceRefs:
      services:
        - clusterName: cluster1
          name: reviews
          namespace: ns1
  trafficShift:
    destinations:
      - kubeService:
          clusterName: cluster2
          name: reviews
          namespace: ns1
          subset:
            version: v3
        weight: 75
      - kubeService:
          clusterName: cluster1
          name: reviews
          namespace: ns1
          subset:
            version: v1
        weight: 15
      - kubeService:
          clusterName: cluster1
          name: reviews
          namespace: ns1
          subset:
            version: v2
        weight: 10

We get this response:

Error from server (Error computing cluster permissions: Traffic policy is not allowed by role default-namespace-admin-role.): error when creating “STDIN”: admission webhook “rbac-webhook.service-mesh-hub.svc” denied the request: Error computing cluster permissions: Traffic policy is not allowed by role default-namespace-admin-role.

We’ve covered a simple (but very common) use case, but as you can see in the Role definition, we can do much more, for example:

Create a role to allow a user to use a specific Virtual Mesh
Create a role to allow a user to use a specific cluster in a Virtual Mesh
Create a role to allow a user to only define Access Policies
Create a role to allow a user to only define Traffic Policies
Create a role to allow a user to only define Failover Services
Create a role to allow a user to only create policies that target the services running in his namespace (but coming from services in any namespace)

Watch this demo to learn more

Get started

We invite you to check out the project and join the community. Solo.io also offers enterprise support for Istio service mesh for those looking to operationalize service mesh environments, request a meeting to learn more here.

Learn more about Gloo Mesh
Read the docs and watch the demos
Request a personalized demo
Questions? Join the community