Bridging a Prometheus Authentication Gap with Gloo Edge Transformations

Prometheus authentication

 

Addressing Prometheus authentication with Gloo Edge transforms

Gloo Edge is a versatile API gateway that provides a rich library of transformation capabilities that can be used to solve a variety of problems, such as Prometheus authentication. Integration issues are often at the heart of what our users want to accomplish: Client X wants to present its requests in a certain format, but Service Y can’t interpret them in that format. Gloo Edge transformation functions are often used to bridge that gap.

That was certainly the case for a Gloo Edge Enterprise customer recently. They approached the Solo field engineering team with this question: “I’m looking for how to allow our customers’ Prometheus instances to scrape a metrics exporter behind our Gloo Edge ext-auth api-key authentication, and am grappling with the constraint that Prometheus has limited options for how it can pass credentials in scrape requests it sends. It can use a bearer token (Authorization: Bearer <secret-value> header) or http basic auth (Authorization: Basic <base64(username:password)> header), but cannot set a custom header (api-key: <secret-value>). Is ext-auth able to check either bearer tokens or basic auth credentials against a set of gloo api-key secrets, similar to how it checks the ApiKeyAuth.header_name header value against those secrets? I don’t see a way in this documentation short of using an ext-auth plugin, but maybe I’m missing something. How would you recommend I do this?”

This is a great question from a customer who has already done some impressive work with their production deployments of Gloo Edge. So let’s unpack this request and establish some context before we proceed to solving the problem.

What’s the problem with Prometheus authentication?

First, let’s establish some Prometheus context from its documentation:  “Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.”

Our customer already has an investment in Gloo Edge Enterprise’s API Key authentication, and they would like to leverage that to allow their end users’ Prometheus instances to scrape Envoy and Gloo metrics from our customer’s Gloo Edge instances.  In other words, they’d like their end users to present a token from their Prometheus clients to Gloo Edge in a way that integrates with its API Key authentication.

But as the original questioner points out, there is a problem with this plan. Prometheus only supports use of bearer tokens or HTTP basic auth in its client scrape configurations.  Our customer would ideally like to present a token to Gloo Edge ext-auth with an api-key header instead.

How can we bridge this gap using Gloo Edge facilities?

Solutions for Prometheus authentication

Custom Ext-Auth Plugin

The original questioner proposes one possible solution: using a Gloo Edge ext-auth plugin to adapt the native Prometheus request, say with a Bearer token, into a request with an api-key header. This is a great option for many use cases, especially where there is a high degree of complexity that exceeds the capability of pre-packaged configuration options. There are excellent blog and documentation resources describing alternative approaches for achieving this, either by building a custom auth server, an ext-auth plugin, or a gRPC passthrough service.

However, with any custom plugin, you must build, maintain, and deploy a separate custom component. That requires extra engineering effort and operational support. While a custom ext-auth plugin could definitely be used here, let’s see if there is a simpler approach to satisfying the requirements of this use case.

Custom WebAssembly Filter

What about a custom WebAssembly (WASM) filter to solve this problem? Many agree that WASM is the future for adding custom extensions in both web browsers and modern proxies like Envoy, which is the foundation of Gloo Edge. In case you’re unfamiliar, WASM began life as a mechanism to add sandboxed custom logic inside web browsers. More recently, its popularity has grown substantially in reverse proxies like Envoy as well. Envoy delivers WASM filters as part of its latest distributions, and Solo.io provides enterprise support for building custom filters using multiple languages, including AssemblyScript, C++, Rust, and TinyGo. For further information, Solo has covered WebAssembly widely in recent months, including blogs here and here, product documentation, and the Hoot podcast.

While WASM is a lighter weight solution than a custom ext-auth plugin, it still requires extra engineering and operational overhead. What if there is an easier way, one that can be expressed in standard Gloo Edge CRDs and without requiring management of custom components?

Transformations

Let’s turn our attention to transformations that can be specified as part of the routing configuration expressed in a Gloo Edge VirtualService.

Let’s review the fundamental problem. We know that the customer’s current API Key auth service is expecting to receive an api-key header with a token value. Further, we know that Prometheus’ client auth can produce Authorization headers with a value like Bearer <token-value>. Can we adapt the header and value that the Prometheus client presents to the format that the auth service expects? The good news is that we can.

Working with Transformations for Prometheus authentication

Configure a Transformation Policy

How can we configure a routing policy to accomplish this transformation?  We’ll specify the solution in a sample VirtualService that must achieve these four outcomes:

  1. Extract the value of the Bearer token from the Authorization header, being careful to extract just the token value and not the entire value of the header.
  2. Create a new api-key header with the extracted token value.
  3. Remove the original Authorization header.
  4. Forward the modified request to the service.

In this post, we’ll focus on just the transformation elements we need to solve this problem. For a more in-depth treatment of this topic, check out the Gloo Edge documentation.

Extract the Bearer Token

The challenge with extracting the Bearer token is that we only want a subset of the Authorization header’s value. The header from the Prometheus client is specified in this format: Authorization: Bearer some-token-value-123. But we only want to extract the token value.

The good news is that Gloo Edge offers a regex feature that allows us to extract only the subset we want.  The extractor stanza below pulls just the value that we want and stores it in the token-extractor variable.

        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1

Create New api-key Header

Having extracted the value we want, we can use an Inja template to inject this into a new request header api-key, which is what the API Key processor in the ext-auth service expects to see.

        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1
              headers:
                api-key:
                  text: '{{ token-extractor }}'

Remove Original Authorization Header

Finally, we’ll add a headerManipulation stanza to remove the original Authorization header. This isn’t strictly required, but it will help ensure that our upstream request is as compact as possible and doesn’t have any unexpected side effects.

        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1
              headers:
                api-key:
                  text: '{{ token-extractor }}'
        headerManipulation:
            requestHeadersToRemove:
            - "Authorization"

Establish Test Environment

If you’d like to follow along in your own environment, all you’ll need is a Kubernetes cluster and associated tools like kubectl, plus an instance of open-source Gloo Edge to complete this guide. Installation instructions are here. If you’re inclined to set this up as our customer did with the Enterprise ext-auth service, then you’ll need a Gloo Edge Enterprise license. You can request a free trial here.

For our testing purposes, we’ll use open-source Gloo Edge and specify a simple VirtualService that runs against the popular httpbin service. This will be sufficient to help us understand the request traffic being passed to the ext-auth service.

Deploy the Httpbin Service

Start by deploying the httpbin service on a Kubernetes cluster.
kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-svc-dpl.yaml

Verify the Upstream

Gloo Edge discovers Kubernetes services automatically and creates a Custom Resource that represents this service, called an Upstream. So, running the glooctl get upstreams command, you should be able to see a new Gloo Edge Upstream default-httpbin-8000, based on the naming convention namespace-serviceName-portNumber:

% glooctl get upstreams default-httpbin-8000
+----------------------+------------+----------+------------------------+
|       UPSTREAM       |    TYPE    |  STATUS  |        DETAILS         |
+----------------------+------------+----------+------------------------+
| default-httpbin-8000 | Kubernetes | Accepted | svc name:      httpbin |
|                      |            |          | svc namespace: default |
|                      |            |          | port:          8000    |
|                      |            |          |                        |
+----------------------+------------+----------+------------------------+

See this documentation to learn more about Upstreams in Gloo Edge.

Create the Virtual Service

Use kubectl to create the following Gloo Edge VirtualService that will route all requests to the new Upstream. A VirtualService represents a policy that manages how external traffic is processed and routed to an Upstream target. To learn more, see the Gloo Edge documentation.
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system
Use this command:
kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-vs-original.yaml
You should observe a result like this:
virtualservice.gateway.solo.io/httpbin-vs created
Run the following glooctl command to confirm that the new Route was accepted by Gloo Edge:
glooctl get virtualservice httpbin-vs
You should see a response similar to this:
% glooctl get virtualservice httpbin-vs
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| httpbin-vs      |              | *       | none | Accepted |                 | / ->                             |
|                 |              |         |      |          |                 | gloo-system.default-httpbin-8000 |
|                 |              |         |      |          |                 | (upstream)                       |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+

Test the Initial Setup

Note that in my case I’m running this in a local KinD cluster, with a port-forward from my Envoy proxy to localhost:8080.
kubectl port-forward -n gloo-system deploy/gateway-proxy 8080 &
proxy_url=http://localhost:8080  # if running in a docker-hosted cluster like KinD
This may be unnecessary for you if you’re running on a “proper” Kubernetes cluster like EKS, GKE, or OpenShift.
proxy_url=$(glooctl proxy url)

You can now access the httpbin endpoints using a curl command. Note that we are emulating the Prometheus client by providing an Authorization header with a Bearer token value.

curl -H "Authorization: Bearer some-token-value-123" ${proxy_url}/get -i

The result should look something like this. Note that the Authorization header is reflected from the httpbin endpoint with no changes at all.

HTTP/1.1 200 OK
server: envoy
date: Thu, 01 Jul 2021 19:41:08 GMT
content-type: application/json
content-length: 291
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Authorization": "Bearer some-token-value-123",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.244.0.28",
  "url": "http://localhost:8080/get"
}

Apply the Transformations

Now we’ll apply the set of transformations that we discussed earlier in the post to our VirtualService.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      options:
        transformations:
          requestTransformation: 
            transformationTemplate:
              extractors:
                token-extractor:
                  header: 'Authorization'
                  regex: 'Bearer (.*)'
                  subgroup: 1
              headers:
                api-key:
                  text: '{{ token-extractor }}'
        headerManipulation:
            requestHeadersToRemove:
            - "Authorization"
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

Apply the transformation changes to the cluster using this command:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-vs-transform.yaml

It should produce this response:

virtualservice.gateway.solo.io/httpbin-vs configured

Finally, we’ll test it out with the same curl command as before.

curl -H "Authorization: Bearer some-token-value-123" ${proxy_url}/get -i

Note that the original Authorization header was not presented to the service.  Instead, it was replaced with an api-key header with a value that contained just the extracted token value, not the entire Bearer value string. That all appears to fit our original objective.

HTTP/1.1 200 OK
server: envoy
date: Thu, 01 Jul 2021 20:41:00 GMT
content-type: application/json
content-length: 278
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Api-Key": "some-token-value-123",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.244.0.28",
  "url": "http://localhost:8080/get"
}

What?!? There’s a Problem?

So far, so good. Right? Wrong!

When we sent this back to the customer, we received some disturbing feedback. This did not solve the problem. They were receiving an error back from their service that the api-key value was missing. How could this be?

The problem was that we forgot a key element of Envoy’s filter chain architecture. Transformations by default are not applied before the call to the ext-auth service. So the original request was being presented to the ext-auth filter before the transformation filter had the chance to apply the new api-key value.

The good news is that we were not the first Gloo Edge users who encountered this problem. Some time ago, the product was enhanced to add a filter for “early” transformation in addition to the “regular” post-ext-auth transformation filter. The figure at the right depicts the Envoy filter chain with early transformation happening near the beginning of request processing.

Apply an early staged transformation is exactly what we need to present the new api-key value to the ext-auth service. See this Gloo Edge documentation if you’d like to explore staged transformations more deeply.

Add an Early-Stage Transformation

Once we had the staged transformation insight, it was easy to modify our VirtualService to apply the policy earlier in the request processing.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      options:
        stagedTransformations:
          early:
            requestTransforms:
              - matcher:
                  prefix: /
                requestTransformation:
                  transformationTemplate:
                    extractors:
                      token-extractor:
                        header: 'Authorization'
                        regex: 'Bearer (.*)'
                        subgroup: 1
                    headers:
                      api-key:
                        text: '{{ token-extractor }}'
        headerManipulation:
            requestHeadersToRemove:
            - "Authorization"
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

Applying this change was easy:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-vs-staged-transform.yaml

This was the response:

virtualservice.gateway.solo.io/httpbin-vs configured

Then we re-ran the curl from before:

curl -H "Authorization: Bearer some-token-value-123" ${proxy_url}/get -i

While the result appeared the same from a curl standpoint in our test sandbox, the customer confirmed that it resolved their problem. The Prometheus client could now authenticate using their existing API Keys with minimal changes to their environment.

HTTP/1.1 200 OK
server: envoy
date: Thu, 01 Jul 2021 21:10:21 GMT
content-type: application/json
content-length: 278
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Api-Key": "some-token-value-123",
    "Host": "localhost:8080",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.244.0.28",
  "url": "http://localhost:8080/get"
}

Learn More

In this blog post, we demonstrated how to use Gloo Edge transformation policies to bridge an authentication scheme gap from a Prometheus client. A customer adapted an incompatible client authentication scheme to integrate with their existing API Key infrastructure by applying less than 20 lines of YAML configuration to a VirtualService.
We accomplished all of this using open-source Gloo Edge and a simple httpbin service to test. If you’d prefer to see this running in a more complete environment with the enterprise-class ext-auth service, then sign up for a trial of Gloo Edge Enterprise to enable that and a lot of other cool features.
You can also check out these resources: