Dynamic Forward Proxy with Gloo Edge

Gloo Edge is a market-leading API gateway in terms of performance and capabilities. Gloo Edge leverages and extends Envoy to provide advanced features such as WAF, advanced authentication and authorization policies, and many enhanced data plane capabilities. On the control plane, one of the key features Gloo Edge provides is that it can seamlessly integrate with its environment to discover the services that are available for routing, especially in cloud-native environments like Kubernetes (other options include Amazon AWS and HashiCorp Consul).

However, there are alternate scenarios to the usual. For example, sometimes there is no service registry. In very dynamic environments where the upstream services cannot be discovered through APIs, you may need to guess–or build–the upstream endpoints based on the Client request. This is commonly called a forward proxy.

Another popular use case is to deploy a forward proxy for all egress traffic. This way, you can observe and control outbound traffic. A common security policy here is rate-limiting and, of course, gathering access logs.

Why do you need a forward proxy?

As stated above, there are two usage models for a forward proxy:

• Routing all the traffic of your local network to an exit node to monitor and control all the egress traffic, or
• Selectively apply it on certain routes, which don’t have a pre-defined destination, and dynamically build the final Host value.

Two of our customers exposed similar use cases: For the first case, their own tenants can manage (deploy/undeploy) services on a public FaaS (Function-as-a-Service) platform; for the second case, end-users can deploy their own logic (transformation pipelines, process automation, etc.). In both instances, the domain names representing the upstream services are not known in advance.

There are a few downsides to such flexibility:

• Since there is no pre-defined Upstream Custom Resource to designate the upstream service, you cannot configure failover policies or client load-balancing.
• DNS resolution is done at runtime. Typically, when a domain name is met for the first time, Envoy will pause the request and synchronously resolve this domain to get the endpoints (IP addresses). Then, these entries are put into a local cache (configurable).

Of course, there are also good reasons why this still makes sense in an API gateway, where you can:

• Easily obtain metrics on the traffic going through the proxy
• Enforce authentication and authorization policies
• Leverage other policies available in Gloo Edge Enterprise, including WAF (Web Application Firewall) or DLP (Data Loss Prevention)

How Gloo Edge embraces dynamic forwarding

Atop these dynamic routing techniques that our multi-tenant-oriented customers love, Gloo Edge can now perform the aforementioned dynamic forwarding in a really nice way.

To start, you need to capture the real destination of the client request. It can simply be the Host header in the most basic setup but it can, of course, be hidden in other client request headers or body parts.

Fortunately, by using Gloo Edge transformation, optionally along with JWT claim extraction, Gloo Edge permits to dynamically build a new header representing the final Host, with a nice templating system. Once this header is created and represents the final destination, you can use it in the RouteAction.

In addition to this dynamic routing logic, you can apply other security policies to the request, like the WAF, JWT authentication, External Authorization, and many more. These options will be applied to the client request before the dynamic forward proxy filter.

Full example

Let’s consider your company is operating a multi-tenant platform, called “mypaas.io”. Your tenant’s environments run behind dedicated domain names and are highly elastic. You want to front these platforms with Gloo Edge, so that you can observe the traffic, and apply both access control rules and also data loss prevention policies.

The code sample below shows a VirtualService configured to perform the following steps:

1. Verify the signature of the client JWT, sent in the Authorization header
2. Extract the claim named tenantId from that JWT and create a new header with it
3. Build up the final domain name using the new header
4. Apply two DLP built-in policies (SSN and ALL_CREDIT_CARDS)

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - 'mypaas.io'
    routes:
      - matchers:
          - prefix: /
        options:
          stagedTransformations:
            regular:
              requestTransforms:
                - matcher:
                    prefix: /
                  requestTransformation:
                    transformationTemplate:
                      headers:
                        x-mypaas-tenant-backend:
                          text: '{{ header("x-mypaas-tenant-id") }}.mypaas.io'
          dlp: # mask some sensible information
            actions:
            - actionType: SSN
            - actionType: ALL_CREDIT_CARDS
        routeAction:
          dynamicForwardProxy:
            autoHostRewriteHeader: "x-mypaas-tenant-backend" # host header will be rewritten to the value of this header
    options:
      jwtStaged:
        afterExtAuth:
          allowMissingOrFailedJwt: true
          providers:
            inline:
              jwks:
                local:
                  key: |
                    -----BEGIN PUBLIC KEY-----
                    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2Es6S529E2T9tzrMeqbb
                    TnkOgdYgOy23dU9SYZCskVotPVV3C9oEuz7q1OkL71PvfUkiqnR3blkpgFvzgCav
                    fzyU+TA2ychxE/ozVwaFKkVhHcm2cng/Bq1cCK1sYSs0lRn8bKqGIayz9OkFazDS
                    9SaasGSn8hWNpXigTS+CFA15xJ/XTMdG22PfPE9037cbXirJeHllPHCiUxKareDR
                    xPSkv/E+ZxM6vPsZm/PAeQHCxA2sglSBQb5+0O/XLaHtsgunXLl7yYK+TcslErGp
                    /Pj6v7dcZosUL3eD66pMSOzgtQddTXPN99RABEYsXFMhjCHbLWtnARQ37AUvPHIN
                    yQIDAQAB
                    -----END PUBLIC KEY-----
              issuer: "mypaas.io"
              tokenSource:
                headers:
                - header: authorization
                  prefix: "Bearer "
              audiences:
              - frontend
              keepToken: true
              claimsToHeaders:
              - claim: tenantId
                header: x-mypaas-tenant-id

Also, you must enable the DFP filter in your Gateway configuration:

kubectl -n gloo-system patch gw/gateway-proxy --type merge -p "
spec:
  httpGateway:
    options:
      dynamicForwardProxy: {}
"

The JWT was generated for demo purposes using the following technique:

brew install mike-engel/jwt-cli/jwt-cli
jwt encode -A RS256 -S @private-key.pem '{
  "sub": "1234567890",
  "name": "John Doe",
  "tenantId": "tenant-1",
  "iat": 1516239022,
  "iss": "mypaas.io",
  "aud": "frontend"
}'
export token="eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiJmcm9udGVuZCIsImlhdCI6MTUxNjIzOTAyMiwiaXNzIjoibXlwYWFzLmlvIiwibmFtZSI6IkpvaG4gRG9lIiwic3ViIjoiMTIzNDU2Nzg5MCIsInRlbmFudElkIjoidGVuYW50LTEifQ.PANYBtfLw1134p8KZpOufXloHwebmYJFI2BwpGUPELVHoSlhVls9Qo-kb-u5dPuS2PrnFdC_HeO9J8xxDa6zymRkj-XiWSloV5VbNdbY_IRoF4y_ZEjgJ_C02FNqLxW5lX0rfZwcZxLDv5hPMQkfXZbhh7rSjD7-CfnRMalfqySf8StdKPes2k-hqosaamyl-d5VhQrAnGDd9dHWf080JzPVaPo9lTHcC_QTCVf-L8jaV00yJ_XZ8SUFcFPEZxxx5dL7gD1ZfatXdemDN6bKBoUvv8KSrRwCOfqk1YUHGxJ3wuHNVAITueTbr3EgAS84e5eYkUnuwvtICRQxCiH_3Q"

And finally, this is how the test is executed:

curl -v $(glooctl proxy url)/ -H "Host: mypaas.io" -H "Authorization: Bearer $token"

In the Envoy logs, here is what we observe:

# the client request comes in like that:
':authority', 'mypaas.io'
':path', '/'
':method', 'GET'
'user-agent', 'curl/7.77.0'
'accept', '*/*'
'authorization', 'Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiJmcm9udGVuZCIsImlhdCI6MTUxNjIzOTAyMiwiaXNzIjoibXlwYWFzLmlvIiwibmFtZSI6IkpvaG4gRG9lIiwic3ViIjoiMTIzNDU2Nzg5MCIsInRlbmFudElkIjoidGVuYW50LTEifQ.PANYBtfLw1134p8KZpOufXloHwebmYJFI2BwpGUPELVHoSlhVls9Qo-kb-u5dPuS2PrnFdC_HeO9J8xxDa6zymRkj-XiWSloV5VbNdbY_IRoF4y_ZEjgJ_C02FNqLxW5lX0rfZwcZxLDv5hPMQkfXZbhh7rSjD7-CfnRMalfqySf8StdKPes2k-hqosaamyl-d5VhQrAnGDd9dHWf080JzPVaPo9lTHcC_QTCVf-L8jaV00yJ_XZ8SUFcFPEZxxx5dL7gD1ZfatXdemDN6bKBoUvv8KSrRwCOfqk1YUHGxJ3wuHNVAITueTbr3EgAS84e5eYkUnuwvtICRQxCiH_3Q'
...
# the JWT filter verifies the signature of the JWT and extracts a claim from it
...
# the transformation filter creates the new header representing the final Host
...
# the dynamic forward filter kicks in and tries to resolve the domain name
[2022-03-16 13:40:22.918][30][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:104] thread local lookup for host 'tenant-1.mypaas.io'
[2022-03-16 13:40:22.918][30][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:125] cache miss for host 'tenant-1.mypaas.io', posting to main thread
[2022-03-16 13:40:22.918][30][debug][forward_proxy] [external/envoy/source/extensions/filters/http/dynamic_forward_proxy/proxy_filter.cc:139] [C23][S6838641320422486301] waiting to load DNS cache entry
[2022-03-16 13:40:22.918][7][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:293] starting main thread resolve for host='tenant-1.mypaas.io' dns='tenant-1.mypaas.io' port='80'
[2022-03-16 13:40:22.918][7][debug][dns] [external/envoy/source/common/network/dns_impl.cc:270] dns resolution for tenant-1.mypaas.io started
[2022-03-16 13:40:22.992][7][debug][dns] [external/envoy/source/common/network/dns_impl.cc:188] dns resolution for tenant-1.mypaas.io completed with status 1
[2022-03-16 13:40:22.992][7][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:316] main thread resolve complete for host 'tenant-1.mypaas.io': []
[2022-03-16 13:40:22.993][30][debug][forward_proxy] [external/envoy/source/extensions/filters/http/dynamic_forward_proxy/proxy_filter.cc:185] [C23][S6838641320422486301] load DNS cache complete, continuing after adding resolved host: tenant-1.mypaas.io
[2022-03-16 13:40:22.993][30][debug][router] [external/envoy/source/common/router/router.cc:457] [C23][S6838641320422486301] cluster 'solo_io_generated_dfp:9830940034953162036' match for URL '/'
[2022-03-16 13:40:22.993][30][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1459] no healthy host for HTTP connection pool
[2022-03-16 13:40:22.993][7][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:400] DNS refresh rate reset for host 'tenant-1.mypaas.io', (failure) refresh rate 60000 ms
...
# the DLP filter is applied on the response
...

The client request is forwarded to tenant-1.mypaas.io.

Learn more

As always, this new feature was implemented to meet customer requests. It took less than one month from the first customer request to the first release! Big shoutout to our great engineering team and to Soloist Kevin Dorosh for delivering this feature.

Find our complete guide for Dynamic Forward Proxy on our documentation website.