Solving a Real-World Information Leakage Problem with WebAssembly and Gloo Edge

Jim Barton | April 9, 2021

Multiple Gloo Edge customers have approached us recently with questions like, “Our product security team wants our applications to remove any response headers from our services that indicate to a potential attacker that we’re using Envoy as the foundation of our API Gateway.  In particular, we’d like to remove the server header and any header like x-envoy-upstream-service-time.  How can Gloo Edge help us with that?”

Another customer made a similar inquiry: “We build VirtualService objects and as part of the route options we have a list of request headers to remove. Things like these:  x-envoy-max-retries, x-envoy-retry-on, x-envoy-retry-grpc-on. Instead of adding over time to this list, is a regex pattern supported? Something like X-Envoy-*?  I didn’t see anything in the documentation.”

Articles like this advocate for scrubbing server responses of any artifacts that might tip a potential bad actor to the details of the server infrastructure that you’re using. They specifically call out the server header as a prime candidate for removal or obfuscation.

We’ll explore these questions using a couple of avenues in this blog post. First, there are some built-in configuration options that come to mind. We’ll walk you step-by-step through how to solve this problem using those open-source features. Second, we’ll leverage one of the newer features of both Gloo Edge and Envoy, by building a custom WebAssembly filter to accomplish the same objective, with a bit more generality. Finally, we’ll compare the throughput and performance of these two approaches.

Prerequisites

You’ll need a Kubernetes cluster and associated tools, plus an instance of Gloo Edge Enterprise to complete this guide. Note that there is a free and open source version of Gloo Edge. It will support the first approach we take to this problem, but only the enterprise version supports the WASM filter integration required for the second approach.

We use Google Kubernetes Engine (GKE) with Kubernetes v1.18.15 to test this guide, although any recent version with any Kubernetes provider should work. If you prefer to work locally instead of on a remote cluster, we’ve have good success with Kind. If you go the Kind route, use these instructions for your installation of Gloo Edge.

We use Gloo Edge Enterprise v1.7.0 as our API gateway. Use this guide if you need to install Gloo Edge Enterprise. If you don’t already have access to Gloo Edge Enterprise, you can request a free trial here.

Expose a Kubernetes Service

We’ll expose the popular httpbin service as the target for this exercise.

Deploy the Service

Start by deploying httpbin on a Kubernetes cluster.
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml

Verify the Upstream

Gloo Edge discovers Kubernetes services automatically and creates a routable target called an Upstream. So, running the glooctl get upstreams command, you should be able to see a new Gloo Edge Upstream default-httpbin-8000, based on the naming convention namespace-serviceName-portNumber:

% glooctl get upstreams default-httpbin-8000
+----------------------+------------+----------+------------------------+
|       UPSTREAM       |    TYPE    |  STATUS  |        DETAILS         |
+----------------------+------------+----------+------------------------+
| default-httpbin-8000 | Kubernetes | Accepted | svc name:      httpbin |
|                      |            |          | svc namespace: default |
|                      |            |          | port:          8000    |
|                      |            |          |                        |
+----------------------+------------+----------+------------------------+

Create the VirtualService

Use kubectl to create the following Gloo Edge VirtualService that will route all requests from any domain “*” to the new Upstream.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system
Run the following glooctl command to confirm that the new Route was accepted by Gloo Edge.
% glooctl get virtualservice httpbin
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| httpbin         | httpbin      | *       | none | Accepted |                 | / ->                             |
|                 |              |         |      |          |                 | gloo-system.default-httpbin-8000 |
|                 |              |         |      |          |                 | (upstream)                       |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+

Test the VirtualService

Use curl to test the httpbin “get” endpoint. Note that the service sends back information mirroring the request that we issued.  Note also that the response headers are included near the beginning of the curl output, including the server and x-envoy-upstream-service-time headers that our security team want suppressed.  The response payload consists of the JSON blob at the end of the curl response.  Note that the “headers” stanza in that JSON do not represent response headers returned to the client.

% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
server: envoy
date: Sun, 04 Apr 2021 01:06:35 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 6

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.19.128",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.104.2.17",
  "url": "http://34.75.19.128/get"
}

Approach #1:  Header Manipulation

Gloo Edge provides a library of header manipulation operations that we can apply to solve this problem. These include both adding and removing request and response headers. We will extend our base VirtualService to name the unwanted headers to be removed.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  displayName: httpbin
  virtualHost:
    domains:
      - '*'
    routes:
      - matchers:
        - prefix: /
        routeAction:
          single:
            upstream:
              name: default-httpbin-8000
              namespace: gloo-system
    options:
      headerManipulation:
        responseHeadersToRemove:
        - "x-envoy-upstream-service-time"
        - "server"

As one of the original customer questions pointed out, a downside to this approach is that it requires exact knowledge of the name of each unwanted header. There are a number of x-envoy-* headers already available, and potentially more could be added in the future. This configuration does not have the flexibility to handle cases like this automatically. This is not a serious limitation, but in the next section we will build a more general filter using WebAssembly.

A Bump in the Road

With the headerManipulation configuration in place, note the response we get back from curl. We see that the server: envoy response header is still returned, despite the fact that we asked Gloo Edge to remove it. This happens because Envoy controls that element of the response, and does not allow the control plane to remove it with this mechanism (or with a WebAssembly filter, for that matter.)

% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
date: Sun, 04 Apr 2021 01:48:52 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true
server: envoy

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.19.128",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.104.2.17",
  "url": "http://34.75.19.128/get"
}

However, there is a solution to the server header problem that spans both the header manipulation and WASM approaches. While still removing the server header using either approach outlined in this blog, also add an option to the Gloo Edge gateway configuration that specifies that the value of the server header (or lack of a value) should PASS_THROUGH to the client.

% kubectl patch gateway -n gloo-system gateway-proxy --type merge -p '{"spec":{"httpGateway":{"options":{"httpConnectionManagerSettings":{"serverHeaderTransformation":"PASS_THROUGH"}}}}}'
gateway.gateway.solo.io/gateway-proxy patched

Note that the resulting call to the httpbin “get” endpoint no longer contains any server header.

% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
date: Mon, 05 Apr 2021 19:04:20 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true

{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "34.75.19.128",
"User-Agent": "curl/7.64.1",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000"
},
"origin": "10.104.2.17",
"url": "http://34.75.19.128/get"
}

There is an alternative approach that would allow you to obfuscate the server header rather than removing it.  Simply patch the gateway to specify the serverName property instead. Envoy will then use that value instead of its default “envoy” value.

Evaluate Performance

We want to conduct a high-level performance test to compare the performance of the native Envoy filter we have configured in this section with the custom WASM filter that we will build next. There is a lot of chatter about WASM performance on the net, and much of it is negative. While this is not by any means the final word on WASM performance — see my colleague Denis Jannot’s blog series that covers Envoy and WASM benchmarking much more thoroughly — this will at least give us a glimpse into what we might expect in the real world.

To accomplish this, we will use the simple but effective web load tester called hey. It is easy to install — brew install hey on MacOS — and just as easy to use. After warming up the cluster, we ran 10,000 requests with the default of 50 client threads against an untuned GKE cluster with three n1-standard-2 VMs. We achieved throughput of 883.86 requests per second with an average response time of 55.5 milliseconds, a p99 of 121.7 milliseconds, and all with no request failures. Later we’ll compare these results with the same workload using our custom WASM filter.

% hey -n 10000 http://34.75.19.128/get

Summary:
  Total:	11.3140 secs
  Slowest:	0.1818 secs
  Fastest:	0.0337 secs
  Average:	0.0555 secs
  Requests/sec:	883.8634

  Total data:	2750000 bytes
  Size/request:	275 bytes

Response time histogram:
  0.034 [1]	|
  0.048 [2665]	|■■■■■■■■■■■■■■■■■■
  0.063 [5902]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.078 [855]	|■■■■■■
  0.093 [242]	|■■
  0.108 [142]	|■
  0.123 [97]	|■
  0.137 [64]	|
  0.152 [21]	|
  0.167 [10]	|
  0.182 [1]	|


Latency distribution:
  10% in 0.0448 secs
  25% in 0.0482 secs
  50% in 0.0520 secs
  75% in 0.0575 secs
  90% in 0.0680 secs
  95% in 0.0811 secs
  99% in 0.1217 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0003 secs, 0.0337 secs, 0.1818 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0005 secs
  resp wait:	0.0552 secs, 0.0336 secs, 0.1818 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0033 secs

Status code distribution:
  [200]	10000 responses

Reset the Configuration

To prepare for part two of this exercise, you’ll want to reset the gateway configuration. First, we’ll replace the Gloo Edge VirtualService that we have now with the original one from the beginning of this post, which does not remove the x-envoy-* headers. Use kubectl to reapply this VirtualService.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - 'glootest.com'
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system
Second, we’ll return the gateway component to its original state as well:
% cat << EOF | kubectl replace -f -
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  httpGateway: {}
  proxyNames:
  - gateway-proxy
  useProxyProto: false
EOF
gateway.gateway.solo.io/gateway-proxy replaced

Approach #2:  Custom WebAssembly Filter

Now let’s explore what it would take to replicate these results using a custom WASM filter using AssemblyScript, a subset of TypeScript designed to build WASM filters.

What is WebAssembly?

WebAssembly (or WASM) began life as a mechanism to add sandboxed custom logic inside web browsers. More recently, its popularity has grown substantially in reverse proxies like Envoy as well. Envoy now delivers WASM filters as part of its latest distributions, and Solo.io provides enterprise support for building custom filters using multiple languages, including AssemblyScript, C++, Rust, and TinyGo. For further information, Solo has covered WebAssembly widely in recent months, including blogs here and here, product documentation, and the Hoot podcast.

Building a WASM Filter

Solo.io’s WebAssembly documentation provides detailed tutorials around building and deploying WASM filters. We will not repeat that material in detail here, but we will provide a summary of the steps for this use case and highlight some of the issues you may encounter if you’re new to WASM. WASM is still in its early days and remains something of a moving target due to the continued influx of innovation. So while the observations present here are current as of this writing, your results may be different.

First, install the free wasme CLI tool and then initialize a new project.

% wasme --version
wasme version 0.0.32

Be sure to choose AssemblyScript as the language to use for this filter.

wasme init ./remove-headers
...
INFO[0007] extracting 1812 bytes to /Users/jibarton/remove-headers

The wasme tool generates a directory tree like this:

% tree .
.
├── assembly
│   ├── index.ts
│   └── tsconfig.json
├── package-lock.json
├── package.json
└── runtime-config.json

1 directory, 5 files

The generated project represents a complete “hello world” WASM example.  We will customize it to fit our use case, which means we will focus on three of the generated files:

  • package.json:  to configure package dependencies
  • runtime-config.json:  to configure a root id for the new project
  • index.ts:  to build the custom WebAssembly code to implement our use case

Update Package Dependencies

The key to determining proper package dependencies is looking for the latest stable set of proxy-runtime dependencies. The proxy-runtime represents the basic runtime services available when building an AssemblyScript filter for Gloo Edge. We start by updating the package.json dependencies to the latest stable proxy-runtime release.  At this writing, that is version 0.1.8. We make a corresponding update to the assembly-script dependency to reflect the version that our release of the proxy-runtime was built against. As stated in the proxy-runtime release note, that is version 0.14.8. We also remove any devDependencies that are supplied with the package.json generated at init time.

This is what our updated package.json looks like:

{
  "scripts": {
    "asbuild:untouched": "asc assembly/index.ts -b build/untouched.wasm --use abort=abort_proc_exit -t build/untouched.wat --validate --sourceMap --debug",
    "asbuild:optimized": "asc assembly/index.ts -b build/optimized.wasm --use abort=abort_proc_exit -t build/optimized.wat --validate --sourceMap --optimize",
    "asbuild": "npm run asbuild:untouched && npm run asbuild:optimized",
    "test": "node tests"
  },
  "dependencies": {
    "@assemblyscript/loader": "^0.14.8",
    "@solo-io/proxy-runtime": "0.1.8",
    "assemblyscript": "^0.14.8"
  }
}

You will likely need to run npm update at this point to ensure that these dependencies are installed on your workstation.

Update Runtime Configuration

This is a small change and optional, but I found it useful.  The runtime-config.json file contains a root id for the project, which the wasme init command provides a default value of add_header. That’s appropriate for the “hello world” use case but not so much for what we’re trying to accomplish here. We made corresponding changes of the value to remove_headers both here and in the index.ts code for the filter itself.

Here is our customized runtime-config.json file with the new root id value.

{
  "type": "envoy_proxy",
  "abiVersions": [
    "v0-541b2c1155fffb15ccde92b8324f3e38f7339ba6",
    "v0-097b7f2e4cc1fb490cc1943d0d633655ac3c522f",
    "v0-4689a30309abf31aee9ae36e73d34b1bb182685f",
    "v0.2.1"
  ],
  "config": {
    "rootIds": [
      "remove_headers"
    ]
  }
}

Write Custom AssemblyScript Code

We spent the bulk of our development effort in creating the code to implement the custom filter. We adapted the AddHeaders produced by wasme init to create the code below.

This is a summary of the changes made:

  • Switch the implemented callback function to onResponseHeaders, since we only need to manipulate the response for our use case. For a full list of available callbacks you can override using Solo’s proxy-runtime, check out the Context class here. In the onResponseHeaders function, we walk through the full list of response headers checking for the presence of any of a list of configured removal tokens. For any of the headers that begins with one of the removal token values, we remove it from the response.
  • We added processing to the constructor to allow for multiple removal tokens to be passed in as a comma-separated list from the external filter configuration. We built this token list in the constructor so as to only build it once when the component is initialized, not on each individual request.
  • We added some log statements in this example, both because they were helpful in debugging and just to demonstrate how it’s done in AssemblyScript.
  • On the last line of this component, we specified the new root id remove_headers to match the customized value in runtime-config.json.
export * from "@solo-io/proxy-runtime/proxy";
import {
  RootContext,
  Context,
  registerRootContext,
  FilterHeadersStatusValues,
  FilterDataStatusValues,
  stream_context,
  log,
  LogLevelValues
} from "@solo-io/proxy-runtime";
class RemoveHeadersRoot extends RootContext {
  createContext(context_id: u32): Context {
    return new RemoveHeader(context_id, this);
  }
}
class RemoveHeader extends Context {

  token_str: string;
  rm_tokens: Array = new Array(10);

  constructor(context_id: u32, root_context: RemoveHeadersRoot) {
    super(context_id, root_context);
    this.token_str = root_context.getConfiguration();
    if (this.token_str != "") {
      // establish array of tokens to remove from response headers
      this.rm_tokens = this.token_str.split(",");
      log(LogLevelValues.debug, "rm-headers: token count: " + this.rm_tokens.length.toString() 
        + " token[0]: " + this.rm_tokens[0]);
    }
  }

  onResponseHeaders(a: u32, end_of_stream: bool): FilterHeadersStatusValues {
    const root_context = this.root_context;
    log(LogLevelValues.trace, "onResponseHeaders called!");
    if (this.token_str == "") {
      log(LogLevelValues.trace, "rm-headers: no config specified - skipping this response");
      return FilterHeadersStatusValues.Continue;
    }

    let hdr_arr = stream_context.headers.response.get_headers();
    let num_hdrs: u32 = hdr_arr.length;
    // search all header keys for the configured tokens and remove the matching headers from response
    for (let i: u32 = 0; i < num_hdrs; i++) {
      let hdr_key: string = String.UTF8.decode(hdr_arr[i].key);
      log(LogLevelValues.debug, "onResponseHeaders processing header: " + hdr_key);
      let num_tokens: u32 = this.rm_tokens.length;
      for (let j: u32 = 0; j < num_tokens; j++) { 
        let rm_token: string = this.rm_tokens[j]; 
        if (hdr_key.startsWith(rm_token)) { 
          stream_context.headers.response.remove(hdr_key); 
          log(LogLevelValues.debug, "onResponseHeaders removed header: " + hdr_key); 
          break; 
        } 
      } 
    } 
    return FilterHeadersStatusValues.Continue; 
  } 
} 
registerRootContext((context_id: u32) => { return new RemoveHeadersRoot(context_id); }, "remove_headers");

Some Design Notes

You may wonder why we chose to remove headers whose names begin with the specified tokens, as opposed to using a more general technique like matching regular expressions. A common theme you’ll find when exploring AssemblyScript is that the runtime libraries are fairly limited compared to full-featured JavaScript or TypeScript.  This makes sense when you think about the overall design objective for these filters to run in more limited, sandbox environments like web browsers or reverse proxies. But it often means that common features like regex may not be supported yet.

You may also wonder from the implementation why we used primitive looping constructs as opposed to more modern JavaScript-style closures. As with regex, this is another case where full language support is not yet available.

Build the Image

Before building the image, you may want to establish a free account at WebAssembly Hub to store your WASM filters.  You can use other image repositories instead, but the wasme CLI already boasts convenient interfaces for managing filters and deploying them to Gloo Edge.

Let’s build the image using wasme. You will of course replace my user account name with your own. If everything works as expected, you should see a transcript something like the one below. You can ignore the benign compilation warnings.

% wasme build assemblyscript -t webassemblyhub.io/jameshbarton/remove-headers:v0.1 .
Building with npm...skipping login
running npm install && npm run asbuild
npm WARN workspace No description
npm WARN workspace No repository field.
npm WARN workspace No license field.

audited 5 packages in 0.881s

1 package is looking for funding
run `npm fund` for details

found 0 vulnerabilities

> @ asbuild /src/workspace
> npm run asbuild:untouched && npm run asbuild:optimized

> @ asbuild:untouched /src/workspace
> asc assembly/index.ts -b build/untouched.wasm --use abort=abort_proc_exit -t build/untouched.wat --validate --sourceMap --debug

WARNING Unknown option '--validate'
WARNING AS201: Conversion from type 'usize' to 'u32' will require an explicit cast when switching between 32/64-bit.

return utoa32(this, radix);
~~~~
in ~lib/number.ts(221,21)

> @ asbuild:optimized /src/workspace
> asc assembly/index.ts -b build/optimized.wasm --use abort=abort_proc_exit -t build/optimized.wat --validate --sourceMap --optimize

WARNING Unknown option '--validate'
WARNING AS201: Conversion from type 'usize' to 'u32' will require an explicit cast when switching between 32/64-bit.

return utoa32(this, radix);
~~~~
in ~lib/number.ts(221,21)

INFO[0014] adding image to cache... filter file=/tmp/wasme725447755/filter.wasm tag="webassemblyhub.io/jameshbarton/remove-headers:v0.1"
INFO[0014] tagged image digest="sha256:04e266f5e2f786fe805dad43c52707dd129bef801c82c02ceb815cc78c6e553c" image="webassemblyhub.io/jameshbarton/remove-headers:v0.1"

The wasme list CLI shows you the images you have cached locally.

% wasme list
NAME                                          TAG  SIZE    SHA      UPDATED
...
webassemblyhub.io/jameshbarton/remove-headers v0.1 19.5 kB 04e266f5 02 Apr 21 17:06 EDT

Push the Image to WebAssembly Hub

Next, deploy the image to WebAssembly Hub.

% wasme push webassemblyhub.io/jameshbarton/remove-headers:v0.1

INFO[0000] Pushing image webassemblyhub.io/jameshbarton/remove-headers:v0.1
INFO[0008] Pushed webassemblyhub.io/jameshbarton/remove-headers:v0.1
INFO[0008] Digest: sha256:c42b2733f85dd5126974f523350f96bf96a1ef7e7f2b3631117acd18c99f3b65

In addition to the wasme CLI, you can also inspect your lists of published images via a web interface.

Deploy the Filter to Gloo Edge

Once you have published the filter to WebAssembly Hub, you can then use wasme to deploy it to your Gloo Edge gateway. Note that the config argument contains a comma-separated list of the header tokens we want to remove from the responses.

% wasme deploy gloo webassemblyhub.io/jameshbarton/remove-headers:v0.1 --id=remove-headers --config "x-envoy,server"
INFO[0003] appending wasm filter filterID=remove-headers
INFO[0005] updated gateway gateway=gloo-system.gateway-proxy
INFO[0005] appending wasm filter filterID=remove-headers
INFO[0005] updated gateway gateway=gloo-system.gateway-proxy-ssl

We will also reapply the gateway-proxy patch that we described in the first section to ensure that Envoy passes through our desired lack of a server header.

% kubectl patch gateway -n gloo-system gateway-proxy --type merge -p '{"spec":{"httpGateway":{"options":{"httpConnectionManagerSettings":{"serverHeaderTransformation":"PASS_THROUGH"}}}}}'
gateway.gateway.solo.io/gateway-proxy patched

Now let’s take a look at the relevant bits of the resulting gateway-proxy configuration. Note the options underneath httpGateway. The wasm configuration ensures that all response headers beginning with either x-envoy or server will be removed. The httpConnectionManagerSettings configuration ensures that Envoy will not override our wishes and return the removed server header to the client anyway.

% kubectl get gateway gateway-proxy -n gloo-system -o yaml
# Output abridged for readability
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy
  namespace: gloo-system
spec:
  httpGateway:
    options:
      httpConnectionManagerSettings:
        serverHeaderTransformation: PASS_THROUGH
      wasm:
        filters:
        - config:
            '@type': type.googleapis.com/google.protobuf.StringValue
            value: x-envoy,server
          image: webassemblyhub.io/jameshbarton/remove-headers:v1.0
          name: remove-headers
          rootId: remove_headers

Note that you can test this for yourself without going through the entire WASM build, test, and deploy process as we have in this post. You can simply run the wasme deploy command as above using this image:  webassemblyhub.io/jameshbarton/remove-headers:v1.0.

Test the WASM Filter

As you can see from the test below, our new configuration removes both the server and x-envoy-* headers from the response as expected.

% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
date: Wed, 07 Apr 2021 14:21:02 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.19.128",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "10.104.2.17",
  "url": "http://34.75.19.128/get"
}

Local Testing Notes

In this section, we moved directly to testing on the remote Kubernetes cluster. In most cases though, you’ll need to spin through a few edit-test-debug cycles before you’re ready to deploy. If that’s the case for you, you’ll want to consider the more efficient option of local testing using a local Envoy instance rather than a remote cluster. This documentation provides the details.

If you do opt for local testing, pay close attention to the embedded Istio version contained in the local Envoy instance. At this writing, it is quite old by default (Istio 1.5). So you may need to override that default to use a more recent Istio image. The example below worked well for much of our local testing.

wasme deploy envoy webassemblyhub.io/jameshbarton/rm-headers:v0.1 --config "x-envoy" --envoy-image docker.io/istio/proxyv2:1.8.1

Note that this advice may not be valid in the future. Expect that WASM support tools will handle this much more gracefully soon.

Evaluate Performance

Finally, we will conduct the same performance evaluation with the WASM filter as we did with the native Envoy filter. And we will use the same environment with the same settings — 10,000 requests with the default maximum of 50 client threads — against a warmed-up but untuned GKE cluster with three n1-standard-2 VMs.

% hey -n 10000 http://34.75.19.128/get

Summary:
  Total:	11.5742 secs
  Slowest:	0.1715 secs
  Fastest:	0.0331 secs
  Average:	0.0568 secs
  Requests/sec:	863.9870

  Total data:	2750000 bytes
  Size/request:	275 bytes

Response time histogram:
  0.033 [1]	|
  0.047 [1486]	|■■■■■■■■■
  0.061 [6370]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.075 [1428]	|■■■■■■■■■
  0.088 [306]	|■■
  0.102 [163]	|■
  0.116 [95]	|■
  0.130 [70]	|
  0.144 [54]	|
  0.158 [14]	|
  0.171 [13]	|


Latency distribution:
  10% in 0.0455 secs
  25% in 0.0490 secs
  50% in 0.0533 secs
  75% in 0.0594 secs
  90% in 0.0698 secs
  95% in 0.0823 secs
  99% in 0.1265 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0003 secs, 0.0331 secs, 0.1715 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0007 secs
  resp wait:	0.0565 secs, 0.0331 secs, 0.1676 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0135 secs

Status code distribution:
  [200]	10000 responses

With the WASM filter, we achieved the same functional results as before with throughput of 863.99 requests per second, an average response time of 56.8 milliseconds and a p99 of 126.5 milliseconds. Comparing these results to the native Envoy filter, the WASM filter throughput lagged by just 2.2% (863.99 vs. 883.86 requests per second), the average response time was 2.3% slower (56.8 vs 55.5 milliseconds), and the p99 lagged by 3.9% (126.5 vs. 121.7 milliseconds.)

As we said before, this is not a rigorous benchmark test and of course, you may see different results. You should always benchmark with your own workloads in your own environment before making deployment decisions.

However, with those caveats, the custom WASM filter works quite well from a performance standpoint next to its native Envoy counterpart.

Watch the Demo

See the demo based on this blog post that was presented at KubeCon Europe 2021.

Learn More

In this blog post, we explored how to solve an information leakage problem using Gloo Edge with both a native Envoy filter and a custom WASM filter. We walked step-by-step through each approach and accomplished the desired results with minimal performance differences between the two approaches.

All of the code used in this guide is available on github.  The final WASM filter image is available for deployment from WebAssembly Hubwebassemblyhub.io/jameshbarton/remove-headers:v1.0

For more information, check out the following resources.

Acknowledgments

A big thank you to Shane O’Donnell for working with me to understand the nuances of building and testing WASM filters with AssemblyScript.

 

Back to Blog