Bridging a Prometheus Authentication Gap with Gloo Edge Transformations

Custom Ext-Auth Plugin

The original questioner proposes one possible solution: using a Gloo Edge ext-auth plugin to adapt the native Prometheus request, say with a Bearer token, into a request with an api-key header. This is a great option for many use cases, especially where there is a high degree of complexity that exceeds the capability of pre-packaged configuration options. There are excellent blog and documentation resources describing alternative approaches for achieving this, either by building a custom auth server, an ext-auth plugin, or a gRPC passthrough service.

However, with any custom plugin, you must build, maintain, and deploy a separate custom component. That requires extra engineering effort and operational support. While a custom ext-auth plugin could definitely be used here, let’s see if there is a simpler approach to satisfying the requirements of this use case.

Custom WebAssembly Filter

What about a custom WebAssembly (WASM) filter to solve this problem? Many agree that WASM is the future for adding custom extensions in both web browsers and modern proxies like Envoy, which is the foundation of Gloo Edge. In case you’re unfamiliar, WASM began life as a mechanism to add sandboxed custom logic inside web browsers. More recently, its popularity has grown substantially in reverse proxies like Envoy as well. Envoy delivers WASM filters as part of its latest distributions, and Solo.io provides enterprise support for building custom filters using multiple languages, including AssemblyScript, C++, Rust, and TinyGo. For further information, Solo has covered WebAssembly widely in recent months, including blogs here and here, product documentation, and the Hoot podcast.

While WASM is a lighter weight solution than a custom ext-auth plugin, it still requires extra engineering and operational overhead. What if there is an easier way, one that can be expressed in standard Gloo Edge CRDs and without requiring management of custom components?

Transformations

Let’s turn our attention to transformations that can be specified as part of the routing configuration expressed in a Gloo Edge VirtualService.

Let’s review the fundamental problem. We know that the customer’s current API Key auth service is expecting to receive an api-key header with a token value. Further, we know that Prometheus’ client auth can produce Authorization headers with a value like Bearer <token-value>. Can we adapt the header and value that the Prometheus client presents to the format that the auth service expects? The good news is that we can.

Working with Transformations for Prometheus authentication

Configure a Transformation Policy

How can we configure a routing policy to accomplish this transformation?  We’ll specify the solution in a sample VirtualService that must achieve these four outcomes:

1. Extract the value of the Bearer token from the Authorization header, being careful to extract just the token value and not the entire value of the header.
2. Create a new api-key header with the extracted token value.
3. Remove the original Authorization header.
4. Forward the modified request to the service.

In this post, we’ll focus on just the transformation elements we need to solve this problem. For a more in-depth treatment of this topic, check out the Gloo Edge documentation.

Extract the Bearer Token

The challenge with extracting the Bearer token is that we only want a subset of the Authorization header’s value. The header from the Prometheus client is specified in this format: Authorization: Bearer some-token-value-123. But we only want to extract the token value.

The good news is that Gloo Edge offers a regex feature that allows us to extract only the subset we want.  The extractor stanza below pulls just the value that we want and stores it in the token-extractor variable.

        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1

Create New api-key Header

Having extracted the value we want, we can use an Inja template to inject this into a new request header api-key, which is what the API Key processor in the ext-auth service expects to see.

        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1
              headers:
                api-key:
                  text: '{{ token-extractor }}'

Remove Original Authorization Header

Finally, we’ll add a headerManipulation stanza to remove the original Authorization header. This isn’t strictly required, but it will help ensure that our upstream request is as compact as possible and doesn’t have any unexpected side effects.

        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1
              headers:
                api-key:
                  text: '{{ token-extractor }}'
        headerManipulation:
            requestHeadersToRemove:
            - "Authorization"

Establish Test Environment

If you’d like to follow along in your own environment, all you’ll need is a Kubernetes cluster and associated tools like kubectl, plus an instance of open-source Gloo Edge to complete this guide. Installation instructions are here. If you’re inclined to set this up as our customer did with the Enterprise ext-auth service, then you’ll need a Gloo Edge Enterprise license. You can request a free trial here.

For our testing purposes, we’ll use open-source Gloo Edge and specify a simple VirtualService that runs against the popular httpbin service. This will be sufficient to help us understand the request traffic being passed to the ext-auth service.

Deploy the Httpbin Service

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-svc-dpl.yaml

Verify the Upstream

% glooctl get upstreams default-httpbin-8000
+----------------------+------------+----------+------------------------+
|       UPSTREAM       |    TYPE    |  STATUS  |        DETAILS         |
+----------------------+------------+----------+------------------------+
| default-httpbin-8000 | Kubernetes | Accepted | svc name:      httpbin |
|                      |            |          | svc namespace: default |
|                      |            |          | port:          8000    |
|                      |            |          |                        |
+----------------------+------------+----------+------------------------+

See this documentation to learn more about Upstreams in Gloo Edge.

Create the Virtual Service

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-vs-original.yaml

virtualservice.gateway.solo.io/httpbin-vs created

glooctl get virtualservice httpbin-vs

% glooctl get virtualservice httpbin-vs
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| httpbin-vs      |              | *       | none | Accepted |                 | / ->                             |
|                 |              |         |      |          |                 | gloo-system.default-httpbin-8000 |
|                 |              |         |      |          |                 | (upstream)                       |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+

Test the Initial Setup

kubectl port-forward -n gloo-system deploy/gateway-proxy 8080 &
proxy_url=http://localhost:8080  # if running in a docker-hosted cluster like KinD

proxy_url=$(glooctl proxy url)

You can now access the httpbin endpoints using a curl command. Note that we are emulating the Prometheus client by providing an Authorization header with a Bearer token value.

curl -H "Authorization: Bearer some-token-value-123" ${proxy_url}/get -i

The result should look something like this. Note that the Authorization header is reflected from the httpbin endpoint with no changes at all.

HTTP/1.1 200 OK
server: envoy
date: Thu, 01 Jul 2021 19:41:08 GMT
content-type: application/json
content-length: 291
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Authorization": "Bearer some-token-value-123",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.244.0.28",
  "url": "http://localhost:8080/get"
}

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      options:
        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1
              headers:
                api-key:
                  text: '{{ token-extractor }}'
        headerManipulation:
            requestHeadersToRemove:
            - "Authorization"
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-vs-transform.yaml

It should produce this response:

virtualservice.gateway.solo.io/httpbin-vs configured

Finally, we’ll test it out with the same curl command as before.

curl -H "Authorization: Bearer some-token-value-123" ${proxy_url}/get -i

Note that the original Authorization header was not presented to the service.  Instead, it was replaced with an api-key header with a value that contained just the extracted token value, not the entire Bearer value string. That all appears to fit our original objective.

HTTP/1.1 200 OK
server: envoy
date: Thu, 01 Jul 2021 20:41:00 GMT
content-type: application/json
content-length: 278
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Api-Key": "some-token-value-123",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.244.0.28",
  "url": "http://localhost:8080/get"
}

What?!? There’s a Problem?

So far, so good. Right? Wrong!

When we sent this back to the customer, we received some disturbing feedback. This did not solve the problem. They were receiving an error back from their service that the api-key value was missing. How could this be?

The problem was that we forgot a key element of Envoy’s filter chain architecture. Transformations by default are not applied before the call to the ext-auth service. So the original request was being presented to the ext-auth filter before the transformation filter had the chance to apply the new api-key value.

The good news is that we were not the first Gloo Edge users who encountered this problem. Some time ago, the product was enhanced to add a filter for “early” transformation in addition to the “regular” post-ext-auth transformation filter. The figure at the right depicts the Envoy filter chain with early transformation happening near the beginning of request processing.

Apply an early staged transformation is exactly what we need to present the new api-key value to the ext-auth service. See this Gloo Edge documentation if you’d like to explore staged transformations more deeply.

Add an Early-Stage Transformation

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      options:
        stagedTransformations:
          early:
            requestTransforms:
              - matcher:
                  prefix: /
                requestTransformation:
                  transformationTemplate:
                    extractors:
                      token-extractor:
                        header: 'Authorization'
                        regex: 'Bearer (.*)'
                        subgroup: 1
                    headers:
                      api-key:
                        text: '{{ token-extractor }}'
        headerManipulation:
            requestHeadersToRemove:
            - "Authorization"
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-vs-staged-transform.yaml

virtualservice.gateway.solo.io/httpbin-vs configured

curl -H "Authorization: Bearer some-token-value-123" ${proxy_url}/get -i

HTTP/1.1 200 OK
server: envoy
date: Thu, 01 Jul 2021 21:10:21 GMT
content-type: application/json
content-length: 278
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Api-Key": "some-token-value-123",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.244.0.28",
  "url": "http://localhost:8080/get"
}