[Tutorial] Advanced mTLS with Gloo Edge

In this article, you will learn how Gloo Edge can enforce a custom security policy.

Gloo Edge

Gloo Edge is the next generation API Gateway built on top of Envoy. This means you get the power of the most famous cloud-native proxy without the hassle of having to understand its inner workings.
All you have to do is expose and secure your services through simple CRDs, graciously documented in our extensive guides section of the Gloo Edge documents.

At the heart of Gloo Edge, Envoy processes requests through different filter chains. Gloo Edge Enterprise (EE) embraces security features and thus comes with many advanced filters such as JWT, OIDC, WAF, advanced CSRF, and many more!

Gloo Edge and mTLS

We will speak about client authentication with certificates, a.k.a. “mutual TLS“. Gloo Edge already supports downstream mTLS, as well as upstream mTLS, and also integration with an existing Istio service-mesh.

If you want to know more about the out-of-the-box support of downstream mTLS, you’ll find a step-by-step tutorial in our dedicated section of the documentation: Configuring downstream mTLS in a Virtual Service. Just keep in mind that you will be able to verify client certs against a CA.

Now, this blog post is about extending this functionality for tailor-made mTLS policies. So, we will build a plugin in this regard.

Luckily, Envoy exposes a wide array of attributes picked from the client certificate. Gloo Edge surfaces these attributes so that you can test them and forward them upstream. But in our case, we want Gloo Edge to allow requests only if the client certificate CN’s value is present in a given list, , declared in a Gloo Custom Resource.

That’s where Gloo Edge shines: you can define your own config templates by leveraging CRDs. In this case, by using a AuthConfig CR.

Here is an example of a Gloo AuthConfig CR, with a custom config: block:

apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: mtls-auth
  namespace: gloo-system
spec:
  configs:
  - pluginAuth:
      name: Mtls
      pluginFileName: Mtls.so
      exportedSymbolName: Plugin
      config:
        HeaderName: "x-forwarded-client-cert"
        AllowedValues:
        - simpleclient
        - foo

That config: block will be read by our plugin. And, as you can already understand, only CN values matching our AllowedValues will be accepted. Great! ?

With this in mind, you’ll now learn how to build this kind of security plugin. From this point, you’ll find a step-by-step tutorial. Let’s start with… the menu!

Table of Contents

Throughout this tutorial, we will:

1. Quick overview of Envoy’s ExtAuthz
2. Generate self-signed certificates for our server and two clients (mTLS standard)
3. Deploy a basic API (backend) on our cluster
4. Deploy Gloo Edge and set up a VirtualService to expose our API
5. Build an ExtAuth plugin
6. Configure this VirtualService so that our mTLS plugin protects it

Envoy’s ExtAuthz

In Envoy’s architecture, the ExtAuthz (i.e., External Authorization) service is where security policies sit. Typically, these policies are in charge of allowing or denying requests.

However, Envoy does not provide this ExtAuthz service OOTB, but some protobuf definitions that your ExtAuthz service must respect (depending on whether you want to apply your policy at the network level or at the HTTP level).

Luckily, Gloo Edge EE comes with such an ExtAuth service, greatly simplifying and supporting many different kinds of Authentication and Authorization. You can even extend it with a Go(lang) plugin. We have a dedicated guide to get you quickly started. 

Your plugin will then run alongside the ExtAuth server.

Prerequisites

• glooctl CLI Installed 
• OpenSSL

A quick way for installing glooctl:

brew install glooctl 
GLOO_VERSION=1.6.3 
glooctl upgrade --release v${GLOO_VERSION}

If you intend to execute this tutorial on your local machine, add the domain to your /etc/hosts file.

GLOO_IP=$(glooctl proxy address | cut -d':' -f1)

cat <<EOF | sudo tee -a /etc/hosts
${GLOO_IP} fakedomain.com
EOF

Step #1 – generate certificates

Let’s first generated self-signed certificates for the server and two sample clients.

mkdir {client,server}-certs

Server-side

cd server-certs

Server CA

openssl genrsa -out ca.key 2048

openssl req -x509 -new -nodes -sha512 -days 365 \
-subj "/C=US/ST=Massachusetts/L=Boston/O=Solo-io/OU=pki/CN=rootca" \
-key ca.key \
-out ca.crt

Server cert

openssl genrsa -out server.key 2048

openssl req -sha512 -new \
-subj "/C=US/ST=Massachusetts/L=Boston/O=Solo-io/OU=pki/CN=fakedomain" \
-key fakedomain.com.key \
-out fakedomain.com.csr

cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=fakedomain.com
EOF

openssl x509 -req -sha512 -days 365 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in fakedomain.com.csr \
-out fakedomain.com.crt

openssl x509 -in fakedomain.com.crt -text

Clients side

cd ../client-certs

Client CA

openssl genrsa -out client-ca.key 2048

openssl req -x509 -new -nodes -sha512 -days 365 \
-subj "/C=US/ST=Massachusetts/L=Boston/O=Solo-io/OU=pki/CN=clientca" \
-key client-ca.key \
-out client-ca.crt

Authorized client cert

CN=simpleclient

openssl genrsa -out simpleclient.key 2048

openssl req -sha512 -new \
-subj "/C=US/ST=Massachusetts/L=Boston/O=Solo-io/OU=pki/CN=simpleclient" \
-key simpleclient.key \
-out simpleclient.csr

cat > v3.ext <<-EOF
keyUsage = critical,digitalSignature
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth
subjectKeyIdentifier = hash
EOF

openssl x509 -req -sha512 -days 365 \
-extfile v3.ext \
-CA client-ca.crt -CAkey client-ca.key -CAcreateserial \
-in simpleclient.csr \
-out simpleclient.crt

openssl x509 -in simpleclient.crt -text

Unauthorized client cert

CN=unauthorized

openssl genrsa -out unauthorized.key 2048

openssl req -sha512 -new \
-subj "/C=US/ST=Massachusetts/L=Boston/O=Solo-io/OU=pki/CN=unauthorized" \
-key unauthorized.key \
-out unauthorized.csr

cat > v3.ext <<-EOF
keyUsage = critical,digitalSignature
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth
subjectKeyIdentifier = hash
EOF

openssl x509 -req -sha512 -days 365 \
-extfile v3.ext \
-CA client-ca.crt -CAkey client-ca.key -CAcreateserial \
-in unauthorized.csr \
-out unauthorized.crt

openssl x509 -in unauthorized.crt -text

Step #2 – deploy a simple backend

We will deploy HTTPBIN as our simple backend API.

kubectl apply -f - <<
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: default
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: default
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
EOF

Step #3 – deploy Gloo Edge and expose our API

The easiest way to deploy Gloo Edge is with its command-line. The advanced way is by using its Helm chart.

First, let’s make it simple:

GLOO_VERSION=1.6.3
glooctl install gateway enterprise --license-key ${LICENSE_KEY} --version ${GLOO_VERSION}
kubectl -n gloo-system wait po --for condition=Ready --timeout -1s --all

Ping us if you need to get started with the Enterprise version.

Behind the scene

Once running, Gloo Edge will discover services running in Kubernetes. For each of these services, Gloo will create an Upstream object. Of course, you can disable this discovery feature or enable it for a given set of namespaces.

Let’s have a look at these Uptreams objects that were created for us:

glooctl get upstream default-httpbin-8000

Having this Upstream created for us, it’s now easy to expose our backend service with a VirtualService:

kubectl apply -f - <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - 'fakedomain.com'
    routes:
      - matchers:
          - prefix: /
        routeAction:
          single:
            upstream:
              name: default-httpbin-8000
              namespace: gloo-system
  sslConfig:
    secretRef:
      name: server-tls-and-client-ca
      namespace: gloo-system
EOF

Finally, let’s add our server cert (+ key) and the client CA to Kubernetes, as a secret:

glooctl create secret tls --name server-tls-and-client-ca \
  --certchain ../server-certs/fakedomain.com.crt \
  --privatekey ../server-certs/fakedomain.com.key \
  --rootca ./client-ca.crt

Test it!

First, check everything is properly configured:

> glooctl check
Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... OK
Checking gateways... OK
Checking proxies... OK
Checking rate limit server... OK
No problems detected.
Skipping Gloo Instance check -- Gloo Federation not detected

Now, let’s test our API without any client cert:

> curl -k $(glooctl proxy url --port https)/get -X GET -H "Host: fakedomain.com"
curl: (35) error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure

With a valid client cert:

> curl -k $(glooctl proxy url --port https)/get -X GET -H "Host: fakedomain.com" --cert simpleclient.crt --key simpleclient.key
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "fakedomain.com",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "192.168.233.80",
  "url": "https://fakedomain.com/get"
}

So far, so good!

Now, let’s bring up some headers with client cert details.

We need to modify our Gateway with some options:

kubectl -n gloo-system patch gw/gateway-proxy-ssl --type merge -p "
spec:
  httpGateway:
    options:
      httpConnectionManagerSettings:
        forwardClientCertDetails: APPEND_FORWARD
        setCurrentClientCertDetails:
          subject: true
"

And see what happens:

> curl -k $(glooctl proxy url --port https)/get -X GET -H "Host: fakedomain.com" --cert simpleclient.crt --key simpleclient.key
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "fakedomain.com",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Forwarded-Client-Cert": "Hash=ef05dd042ac45bcc27e5c232c93829cb62165a4bf058f8dcf165f326b7c9f939;Subject=\"CN=simpleclient,OU=clients,O=Solo-io,L=Boston,C=US\""
  },
  "origin": "192.168.233.80",
  "url": "https://fakedomain.com/get"
}

And voilà! A new header named X-Forwarded-Client-Cert popped up with some details picked up from the client cert.

Now, we want to set up some custom logic for accepting or denying requests.

Step #4 – build an ExtAuth plugin

As mentioned earlier in this blog entry, an ExtAuth plugin allows you to implement some business logic for Authorization concerns.

There are several articles & blog entries to help you with that. So let’s make it short this time.

Here is how requests coming to the ExtAuth service are processed by the Go plugin:

func (c *MtlsAuthService) Authorize(ctx context.Context, request *api.AuthorizationRequest) (*api.AuthorizationResponse, error) {
	// ...
	for key, value := range request.CheckRequest.GetAttributes().GetRequest().GetHttp().GetHeaders() {
		if key == c.HeaderName {
			logger(ctx).Infow("Found required header, checking value.", "header", key, "value", value)

			cn := ""
			var cnRegexp = regexp.MustCompile(`.*Subject=(.*CN=(?P<CN>[a-zA-Z0-9:-_*.]+).*).*`)
			stringSubmatch := cnRegexp.FindStringSubmatch(value)
			for i, name := range cnRegexp.SubexpNames() {
				if name == "CN" {
					cn = stringSubmatch[i]
				}
			}

			if _, ok := c.AllowedValues[cn]; ok {
				logger(ctx).Infow("Header value match. Allowing request.")
				response := api.AuthorizedResponse()

				// Append extra header
				response.CheckResponse.HttpResponse = &envoy_service_auth_v3.CheckResponse_OkResponse{
					OkResponse: &envoy_service_auth_v3.OkHttpResponse{
						Headers: []*envoy_config_core_v3.HeaderValueOption{{
							Header: &envoy_config_core_v3.HeaderValue{
								Key:   "mtls-allowed-client",
								Value: "true",
							},
						},
						{
							Header: &envoy_config_core_v3.HeaderValue{
								Key:   "mtls-subject",
								Value: cn,
							},
						}},
					},
				}
				return response, nil
			}
			logger(ctx).Infow("Header value does not match allowed values, denying access.")
			return api.UnauthorizedResponse(), nil
		}
	}
	logger(ctx).Infow("Required header not found, denying access")
	return api.UnauthorizedResponse(), nil
}

And the way we’ll configure this policy, thanks to an AuthConfig CRD:

apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: mtls-auth
  namespace: gloo-system
spec:
  configs:
  - pluginAuth:
      name: Mtls
      pluginFileName: Mtls.so
      exportedSymbolName: Plugin
      config:
        HeaderName: "x-forwarded-client-cert"
        AllowedValues:
        - simpleclient
        - foo

So, here we take advantage of the AuthConfig CRD to push some config parameters to our plugin. It’s what you can see under the config: attribute.

Only client certs having the CN field equal to simpleclient or foo will be accepted.

Well, let’s patch our Gloo Edge deployment so that the ExtAuth server starts with the plugin:

cat << EOF > extauth-plugin-values.yaml
global:
  extensions:
    extAuth:
      plugins:
        mtls:
          image:
            repository: ext-auth-plugin-mtls
            registry: docker.io/pileenretard
            pullPolicy: Always
            tag: 1.6.3
EOF

# first check you won't break it all... (btw. helm diff is a great plugin!)
helm diff upgrade gloo glooe/gloo-ee \
    --namespace gloo-system \
    --version ${GLOO_VERSION} \
    --set-string license_key=${LICENSE_KEY} \
    -f extauth-plugin-values.yaml 
    
# upgrade
helm upgrade gloo glooe/gloo-ee \
    --namespace gloo-system \
    --version ${GLOO_VERSION} \
    --set-string license_key=${LICENSE_KEY} \
    -f extauth-plugin-values.yaml

Check the extauth pod is running, then apply the AuthConfig, as shown above.

Step #5 – configure the VirtualService

Final step, let’s tell our VirtualService to use our AuthConfig definition:

kubectl -n gloo-system patch vs/httpbin --type merge -p "
spec:
  virtualHost:
    domains:
      - 'fakedomain.com'
    options:
      extauth:
        configRef:
          name: mtls-auth
          namespace: gloo-system
"

One last check:

> glooctl check                                                                                         [18:21:28]
Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... OK
Checking gateways... OK
Checking proxies... OK
Checking rate limit server... OK
No problems detected.
Skipping Gloo Instance check -- Gloo Federation not detected

We are ready to go!

Test it!

For testing purpose, we’ll set the ExtAuth pod log level to debug:

kubectl -n gloo-system port-forward deployment/extauth 9091:9091

Then click the button next to debug:

Test with an allowed client CN (CN=simpleclient)

> curl -k $(glooctl proxy url --port https)/get -X GET -H "Host: fakedomain.com" --cert simpleclient.crt --key simpleclient.key
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "fakedomain.com",
    "Mtls-Allowed-Client": "true",
    "Mtls-Subject": "simpleclient",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Forwarded-Client-Cert": "Hash=ef05dd042ac45bcc27e5c232c93829cb62165a4bf058f8dcf165f326b7c9f939;Subject=\"CN=simpleclient,OU=clients,O=Solo-io,L=Boston,C=US\""
  },
  "origin": "192.168.233.80",
  "url": "https://fakedomain.com/get"
}

Furthermore, as you can see, the client’s CN was pushed to the upstream server in a new header named Mtls-Subject.

Test with another client CN, not being part of the allowed list (CN=unauthorized)

> curl -k $(glooctl proxy url --port https)/get -X GET -H "Host: fakedomain.com" --cert unauthorized.crt --key unauthorized.key -I
HTTP/2 403
date: Mon, 25 Jan 2021 17:36:49 GMT
server: envoy

The client is returned a 403 status code.

If we take a look at the ExtAuth pod, we can see logs from our plugin:

{
  "level": "info",
  "ts": 1611596165.545395,
  "logger": "ext-auth.ext-auth-service.mtls_plugin",
  "msg": "Found required header, checking value.",
  "version": "undefined",
  "header": "x-forwarded-client-cert",
  "value": "Hash=0a48da355f8105354367961c2bd6806a7bf0e5271c2f336686c78afb4fd8afc0;Subject=\"CN=unauthorized,OU=pki,O=Solo-io,L=Boston,ST=Massachusetts,C=US\""
}
{
  "level": "info",
  "ts": 1611596165.545546,
  "logger": "ext-auth.ext-auth-service.mtls_plugin",
  "msg": "Header value does not match allowed values, denying access.",
  "version": "undefined"
}
{
  "level": "debug",
  "ts": 1611596165.5455935,
  "logger": "ext-auth.ext-auth-service",
  "msg": "Access denied by auth authService",
  "version": "undefined",
  "authService": "Mtls"
}

That’s all for today!

Wrapping up

We’ve gone through deploying a backend service, building a custom ExtAuth plugin, and applying this plugin to our VirtualService.

If you don’t want to bother with ExtAuth plugins compilation, etc., there is now another way to go, called PassThrough Auth. Basically, this is an independent service that communicates with the ExtAuth pod over gRPC.

On the one hand, you don’t have to recompile your Auth plugin every time you upgrade Gloo; on the other hand, you won’t have the opportunity to configure your plugin with AuthConfig CRs.

Learn More

You can request a free trial of Gloo Edge today here. To connect, join the #gloo-edge and #gloo-enterprise channels in the Solo.io Slack.