Solving a Real-World Information Leakage Problem with WebAssembly and Gloo Edge
Multiple Gloo Edge customers have approached us recently with questions like, “Our product security team wants our applications to remove any response headers from our services that indicate to a potential attacker that we’re using Envoy as the foundation of our API Gateway. In particular, we’d like to remove the server header and any header like x-envoy-upstream-service-time. How can Gloo Edge help us with that?”
Another customer made a similar inquiry: “We build VirtualService objects and as part of the route options we have a list of request headers to remove. Things like these: x-envoy-max-retries, x-envoy-retry-on, x-envoy-retry-grpc-on. Instead of adding over time to this list, is a regex pattern supported? Something like X-Envoy-*? I didn’t see anything in the documentation.”
Articles like this advocate for scrubbing server responses of any artifacts that might tip a potential bad actor to the details of the server infrastructure that you’re using. They specifically call out the server header as a prime candidate for removal or obfuscation.
We’ll explore these questions using a couple of avenues in this blog post. First, there are some built-in configuration options that come to mind. We’ll walk you step-by-step through how to solve this problem using those open-source features. Second, we’ll leverage one of the newer features of both Gloo Edge and Envoy, by building a custom WebAssembly filter to accomplish the same objective, with a bit more generality. Finally, we’ll compare the throughput and performance of these two approaches.
Prerequisites
You’ll need a Kubernetes cluster and associated tools, plus an instance of Gloo Edge Enterprise to complete this guide. Note that there is a free and open source version of Gloo Edge. It will support the first approach we take to this problem, but only the enterprise version supports the WASM filter integration required for the second approach.
We use Google Kubernetes Engine (GKE) with Kubernetes v1.18.15 to test this guide, although any recent version with any Kubernetes provider should work. If you prefer to work locally instead of on a remote cluster, we’ve have good success with Kind. If you go the Kind route, use these instructions for your installation of Gloo Edge.
We use Gloo Edge Enterprise v1.7.0 as our API gateway. Use this guide if you need to install Gloo Edge Enterprise. If you don’t already have access to Gloo Edge Enterprise, you can request a free trial here.
Expose a Kubernetes Service
We’ll expose the popular httpbin service as the target for this exercise.
Deploy the Service
httpbin on a Kubernetes cluster.kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml
Verify the Upstream
Gloo Edge discovers Kubernetes services automatically and creates a routable target called an Upstream. So, running the glooctl get upstreams command, you should be able to see a new Gloo Edge Upstream default-httpbin-8000, based on the naming convention namespace-serviceName-portNumber:
% glooctl get upstreams default-httpbin-8000 +----------------------+------------+----------+------------------------+ | UPSTREAM | TYPE | STATUS | DETAILS | +----------------------+------------+----------+------------------------+ | default-httpbin-8000 | Kubernetes | Accepted | svc name: httpbin | | | | | svc namespace: default | | | | | port: 8000 | | | | | | +----------------------+------------+----------+------------------------+
Create the VirtualService
Use kubectl to create the following Gloo Edge VirtualService that will route all requests from any domain “*” to the new Upstream.
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: httpbin
namespace: gloo-system
spec:
virtualHost:
domains:
- '*'
routes:
- matchers:
- prefix: /
routeAction:
single:
upstream:
name: default-httpbin-8000
namespace: gloo-system
glooctl command to confirm that the new Route was accepted by Gloo Edge.% glooctl get virtualservice httpbin +-----------------+--------------+---------+------+----------+-----------------+----------------------------------+ | VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES | +-----------------+--------------+---------+------+----------+-----------------+----------------------------------+ | httpbin | httpbin | * | none | Accepted | | / -> | | | | | | | | gloo-system.default-httpbin-8000 | | | | | | | | (upstream) | +-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
Test the VirtualService
Use curl to test the httpbin “get” endpoint. Note that the service sends back information mirroring the request that we issued. Note also that the response headers are included near the beginning of the curl output, including the server and x-envoy-upstream-service-time headers that our security team want suppressed. The response payload consists of the JSON blob at the end of the curl response. Note that the “headers” stanza in that JSON do not represent response headers returned to the client.
% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
server: envoy
date: Sun, 04 Apr 2021 01:06:35 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 6
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "34.75.19.128",
"User-Agent": "curl/7.64.1",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000"
},
"origin": "10.104.2.17",
"url": "http://34.75.19.128/get"
}
Approach #1: Header Manipulation
Gloo Edge provides a library of header manipulation operations that we can apply to solve this problem. These include both adding and removing request and response headers. We will extend our base VirtualService to name the unwanted headers to be removed.
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: httpbin
namespace: gloo-system
spec:
displayName: httpbin
virtualHost:
domains:
- '*'
routes:
- matchers:
- prefix: /
routeAction:
single:
upstream:
name: default-httpbin-8000
namespace: gloo-system
options:
headerManipulation:
responseHeadersToRemove:
- "x-envoy-upstream-service-time"
- "server"
As one of the original customer questions pointed out, a downside to this approach is that it requires exact knowledge of the name of each unwanted header. There are a number of x-envoy-* headers already available, and potentially more could be added in the future. This configuration does not have the flexibility to handle cases like this automatically. This is not a serious limitation, but in the next section we will build a more general filter using WebAssembly.
A Bump in the Road
With the headerManipulation configuration in place, note the response we get back from curl. We see that the server: envoy response header is still returned, despite the fact that we asked Gloo Edge to remove it. This happens because Envoy controls that element of the response, and does not allow the control plane to remove it with this mechanism (or with a WebAssembly filter, for that matter.)
% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
date: Sun, 04 Apr 2021 01:48:52 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true
server: envoy
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "34.75.19.128",
"User-Agent": "curl/7.64.1",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000"
},
"origin": "10.104.2.17",
"url": "http://34.75.19.128/get"
}
However, there is a solution to the server header problem that spans both the header manipulation and WASM approaches. While still removing the server header using either approach outlined in this blog, also add an option to the Gloo Edge gateway configuration that specifies that the value of the server header (or lack of a value) should PASS_THROUGH to the client.
% kubectl patch gateway -n gloo-system gateway-proxy --type merge -p '{"spec":{"httpGateway":{"options":{"httpConnectionManagerSettings":{"serverHeaderTransformation":"PASS_THROUGH"}}}}}'
gateway.gateway.solo.io/gateway-proxy patched
Note that the resulting call to the httpbin “get” endpoint no longer contains any server header.
% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
date: Mon, 05 Apr 2021 19:04:20 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "34.75.19.128",
"User-Agent": "curl/7.64.1",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000"
},
"origin": "10.104.2.17",
"url": "http://34.75.19.128/get"
}
There is an alternative approach that would allow you to obfuscate the server header rather than removing it. Simply patch the gateway to specify the serverName property instead. Envoy will then use that value instead of its default “envoy” value.
Evaluate Performance
We want to conduct a high-level performance test to compare the performance of the native Envoy filter we have configured in this section with the custom WASM filter that we will build next. There is a lot of chatter about WASM performance on the net, and much of it is negative. While this is not by any means the final word on WASM performance — see my colleague Denis Jannot’s blog series that covers Envoy and WASM benchmarking much more thoroughly — this will at least give us a glimpse into what we might expect in the real world.
To accomplish this, we will use the simple but effective web load tester called hey. It is easy to install — brew install hey on MacOS — and just as easy to use. After warming up the cluster, we ran 10,000 requests with the default of 50 client threads against an untuned GKE cluster with three n1-standard-2 VMs. We achieved throughput of 883.86 requests per second with an average response time of 55.5 milliseconds, a p99 of 121.7 milliseconds, and all with no request failures. Later we’ll compare these results with the same workload using our custom WASM filter.
% hey -n 10000 http://34.75.19.128/get Summary: Total: 11.3140 secs Slowest: 0.1818 secs Fastest: 0.0337 secs Average: 0.0555 secs Requests/sec: 883.8634 Total data: 2750000 bytes Size/request: 275 bytes Response time histogram: 0.034 [1] | 0.048 [2665] |■■■■■■■■■■■■■■■■■■ 0.063 [5902] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.078 [855] |■■■■■■ 0.093 [242] |■■ 0.108 [142] |■ 0.123 [97] |■ 0.137 [64] | 0.152 [21] | 0.167 [10] | 0.182 [1] | Latency distribution: 10% in 0.0448 secs 25% in 0.0482 secs 50% in 0.0520 secs 75% in 0.0575 secs 90% in 0.0680 secs 95% in 0.0811 secs 99% in 0.1217 secs Details (average, fastest, slowest): DNS+dialup: 0.0003 secs, 0.0337 secs, 0.1818 secs DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs req write: 0.0000 secs, 0.0000 secs, 0.0005 secs resp wait: 0.0552 secs, 0.0336 secs, 0.1818 secs resp read: 0.0000 secs, 0.0000 secs, 0.0033 secs Status code distribution: [200] 10000 responses
Reset the Configuration
To prepare for part two of this exercise, you’ll want to reset the gateway configuration. First, we’ll replace the Gloo Edge VirtualService that we have now with the original one from the beginning of this post, which does not remove the x-envoy-* headers. Use kubectl to reapply this VirtualService.
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: httpbin
namespace: gloo-system
spec:
virtualHost:
domains:
- 'glootest.com'
routes:
- matchers:
- prefix: /
routeAction:
single:
upstream:
name: default-httpbin-8000
namespace: gloo-system
gateway component to its original state as well:% cat << EOF | kubectl replace -f -
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
labels:
app: gloo
name: gateway-proxy
namespace: gloo-system
spec:
bindAddress: '::'
bindPort: 8080
httpGateway: {}
proxyNames:
- gateway-proxy
useProxyProto: false
EOF
gateway.gateway.solo.io/gateway-proxy replaced
Approach #2: Custom WebAssembly Filter
Now let’s explore what it would take to replicate these results using a custom WASM filter using AssemblyScript, a subset of TypeScript designed to build WASM filters.
What is WebAssembly?
Building a WASM Filter
Solo.io’s WebAssembly documentation provides detailed tutorials around building and deploying WASM filters. We will not repeat that material in detail here, but we will provide a summary of the steps for this use case and highlight some of the issues you may encounter if you’re new to WASM. WASM is still in its early days and remains something of a moving target due to the continued influx of innovation. So while the observations present here are current as of this writing, your results may be different.
First, install the free wasme CLI tool and then initialize a new project.
% wasme --version wasme version 0.0.32
Be sure to choose AssemblyScript as the language to use for this filter.
wasme init ./remove-headers ... INFO[0007] extracting 1812 bytes to /Users/jibarton/remove-headers
The wasme tool generates a directory tree like this:
% tree . . ├── assembly │ ├── index.ts │ └── tsconfig.json ├── package-lock.json ├── package.json └── runtime-config.json 1 directory, 5 files
The generated project represents a complete “hello world” WASM example. We will customize it to fit our use case, which means we will focus on three of the generated files:
package.json: to configure package dependenciesruntime-config.json: to configure a root id for the new projectindex.ts: to build the custom WebAssembly code to implement our use case
Update Package Dependencies
The key to determining proper package dependencies is looking for the latest stable set of proxy-runtime dependencies. The proxy-runtime represents the basic runtime services available when building an AssemblyScript filter for Gloo Edge. We start by updating the package.json dependencies to the latest stable proxy-runtime release. At this writing, that is version 0.1.8. We make a corresponding update to the assembly-script dependency to reflect the version that our release of the proxy-runtime was built against. As stated in the proxy-runtime release note, that is version 0.14.8. We also remove any devDependencies that are supplied with the package.json generated at init time.
This is what our updated package.json looks like:
{
"scripts": {
"asbuild:untouched": "asc assembly/index.ts -b build/untouched.wasm --use abort=abort_proc_exit -t build/untouched.wat --validate --sourceMap --debug",
"asbuild:optimized": "asc assembly/index.ts -b build/optimized.wasm --use abort=abort_proc_exit -t build/optimized.wat --validate --sourceMap --optimize",
"asbuild": "npm run asbuild:untouched && npm run asbuild:optimized",
"test": "node tests"
},
"dependencies": {
"@assemblyscript/loader": "^0.14.8",
"@solo-io/proxy-runtime": "0.1.8",
"assemblyscript": "^0.14.8"
}
}
You will likely need to run npm update at this point to ensure that these dependencies are installed on your workstation.
Update Runtime Configuration
This is a small change and optional, but I found it useful. The runtime-config.json file contains a root id for the project, which the wasme init command provides a default value of add_header. That’s appropriate for the “hello world” use case but not so much for what we’re trying to accomplish here. We made corresponding changes of the value to remove_headers both here and in the index.ts code for the filter itself.
Here is our customized runtime-config.json file with the new root id value.
{
"type": "envoy_proxy",
"abiVersions": [
"v0-541b2c1155fffb15ccde92b8324f3e38f7339ba6",
"v0-097b7f2e4cc1fb490cc1943d0d633655ac3c522f",
"v0-4689a30309abf31aee9ae36e73d34b1bb182685f",
"v0.2.1"
],
"config": {
"rootIds": [
"remove_headers"
]
}
}
Write Custom AssemblyScript Code
We spent the bulk of our development effort in creating the code to implement the custom filter. We adapted the AddHeaders produced by wasme init to create the code below.
This is a summary of the changes made:
- Switch the implemented callback function to
onResponseHeaders, since we only need to manipulate the response for our use case. For a full list of available callbacks you can override using Solo’sproxy-runtime, check out theContextclass here. In the onResponseHeaders function, we walk through the full list of response headers checking for the presence of any of a list of configured removal tokens. For any of the headers that begins with one of the removal token values, we remove it from the response. - We added processing to the constructor to allow for multiple removal tokens to be passed in as a comma-separated list from the external filter configuration. We built this token list in the constructor so as to only build it once when the component is initialized, not on each individual request.
- We added some log statements in this example, both because they were helpful in debugging and just to demonstrate how it’s done in AssemblyScript.
- On the last line of this component, we specified the new root id
remove_headersto match the customized value inruntime-config.json.
export * from "@solo-io/proxy-runtime/proxy";
import {
RootContext,
Context,
registerRootContext,
FilterHeadersStatusValues,
FilterDataStatusValues,
stream_context,
log,
LogLevelValues
} from "@solo-io/proxy-runtime";
class RemoveHeadersRoot extends RootContext {
createContext(context_id: u32): Context {
return new RemoveHeader(context_id, this);
}
}
class RemoveHeader extends Context {
token_str: string;
rm_tokens: Array = new Array(10);
constructor(context_id: u32, root_context: RemoveHeadersRoot) {
super(context_id, root_context);
this.token_str = root_context.getConfiguration();
if (this.token_str != "") {
// establish array of tokens to remove from response headers
this.rm_tokens = this.token_str.split(",");
log(LogLevelValues.debug, "rm-headers: token count: " + this.rm_tokens.length.toString()
+ " token[0]: " + this.rm_tokens[0]);
}
}
onResponseHeaders(a: u32, end_of_stream: bool): FilterHeadersStatusValues {
const root_context = this.root_context;
log(LogLevelValues.trace, "onResponseHeaders called!");
if (this.token_str == "") {
log(LogLevelValues.trace, "rm-headers: no config specified - skipping this response");
return FilterHeadersStatusValues.Continue;
}
let hdr_arr = stream_context.headers.response.get_headers();
let num_hdrs: u32 = hdr_arr.length;
// search all header keys for the configured tokens and remove the matching headers from response
for (let i: u32 = 0; i < num_hdrs; i++) {
let hdr_key: string = String.UTF8.decode(hdr_arr[i].key);
log(LogLevelValues.debug, "onResponseHeaders processing header: " + hdr_key);
let num_tokens: u32 = this.rm_tokens.length;
for (let j: u32 = 0; j < num_tokens; j++) {
let rm_token: string = this.rm_tokens[j];
if (hdr_key.startsWith(rm_token)) {
stream_context.headers.response.remove(hdr_key);
log(LogLevelValues.debug, "onResponseHeaders removed header: " + hdr_key);
break;
}
}
}
return FilterHeadersStatusValues.Continue;
}
}
registerRootContext((context_id: u32) => { return new RemoveHeadersRoot(context_id); }, "remove_headers");
Some Design Notes
You may wonder why we chose to remove headers whose names begin with the specified tokens, as opposed to using a more general technique like matching regular expressions. A common theme you’ll find when exploring AssemblyScript is that the runtime libraries are fairly limited compared to full-featured JavaScript or TypeScript. This makes sense when you think about the overall design objective for these filters to run in more limited, sandbox environments like web browsers or reverse proxies. But it often means that common features like regex may not be supported yet.
You may also wonder from the implementation why we used primitive looping constructs as opposed to more modern JavaScript-style closures. As with regex, this is another case where full language support is not yet available.
Build the Image
Before building the image, you may want to establish a free account at WebAssembly Hub to store your WASM filters. You can use other image repositories instead, but the wasme CLI already boasts convenient interfaces for managing filters and deploying them to Gloo Edge.
Let’s build the image using wasme. You will of course replace my user account name with your own. If everything works as expected, you should see a transcript something like the one below. You can ignore the benign compilation warnings.
% wasme build assemblyscript -t webassemblyhub.io/jameshbarton/remove-headers:v0.1 . Building with npm...skipping login running npm install && npm run asbuild npm WARN workspace No description npm WARN workspace No repository field. npm WARN workspace No license field. audited 5 packages in 0.881s 1 package is looking for funding run `npm fund` for details found 0 vulnerabilities > @ asbuild /src/workspace > npm run asbuild:untouched && npm run asbuild:optimized > @ asbuild:untouched /src/workspace > asc assembly/index.ts -b build/untouched.wasm --use abort=abort_proc_exit -t build/untouched.wat --validate --sourceMap --debug WARNING Unknown option '--validate' WARNING AS201: Conversion from type 'usize' to 'u32' will require an explicit cast when switching between 32/64-bit. return utoa32(this, radix); ~~~~ in ~lib/number.ts(221,21) > @ asbuild:optimized /src/workspace > asc assembly/index.ts -b build/optimized.wasm --use abort=abort_proc_exit -t build/optimized.wat --validate --sourceMap --optimize WARNING Unknown option '--validate' WARNING AS201: Conversion from type 'usize' to 'u32' will require an explicit cast when switching between 32/64-bit. return utoa32(this, radix); ~~~~ in ~lib/number.ts(221,21) INFO[0014] adding image to cache... filter file=/tmp/wasme725447755/filter.wasm tag="webassemblyhub.io/jameshbarton/remove-headers:v0.1" INFO[0014] tagged image digest="sha256:04e266f5e2f786fe805dad43c52707dd129bef801c82c02ceb815cc78c6e553c" image="webassemblyhub.io/jameshbarton/remove-headers:v0.1"
The wasme list CLI shows you the images you have cached locally.
% wasme list NAME TAG SIZE SHA UPDATED ... webassemblyhub.io/jameshbarton/remove-headers v0.1 19.5 kB 04e266f5 02 Apr 21 17:06 EDT
Push the Image to WebAssembly Hub
Next, deploy the image to WebAssembly Hub.
% wasme push webassemblyhub.io/jameshbarton/remove-headers:v0.1 INFO[0000] Pushing image webassemblyhub.io/jameshbarton/remove-headers:v0.1 INFO[0008] Pushed webassemblyhub.io/jameshbarton/remove-headers:v0.1 INFO[0008] Digest: sha256:c42b2733f85dd5126974f523350f96bf96a1ef7e7f2b3631117acd18c99f3b65
In addition to the wasme CLI, you can also inspect your lists of published images via a web interface.
Deploy the Filter to Gloo Edge
Once you have published the filter to WebAssembly Hub, you can then use wasme to deploy it to your Gloo Edge gateway. Note that the config argument contains a comma-separated list of the header tokens we want to remove from the responses.
% wasme deploy gloo webassemblyhub.io/jameshbarton/remove-headers:v0.1 --id=remove-headers --config "x-envoy,server" INFO[0003] appending wasm filter filterID=remove-headers INFO[0005] updated gateway gateway=gloo-system.gateway-proxy INFO[0005] appending wasm filter filterID=remove-headers INFO[0005] updated gateway gateway=gloo-system.gateway-proxy-ssl
We will also reapply the gateway-proxy patch that we described in the first section to ensure that Envoy passes through our desired lack of a server header.
% kubectl patch gateway -n gloo-system gateway-proxy --type merge -p '{"spec":{"httpGateway":{"options":{"httpConnectionManagerSettings":{"serverHeaderTransformation":"PASS_THROUGH"}}}}}'
gateway.gateway.solo.io/gateway-proxy patched
Now let’s take a look at the relevant bits of the resulting gateway-proxy configuration. Note the options underneath httpGateway. The wasm configuration ensures that all response headers beginning with either x-envoy or server will be removed. The httpConnectionManagerSettings configuration ensures that Envoy will not override our wishes and return the removed server header to the client anyway.
% kubectl get gateway gateway-proxy -n gloo-system -o yaml
# Output abridged for readability
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
name: gateway-proxy
namespace: gloo-system
spec:
httpGateway:
options:
httpConnectionManagerSettings:
serverHeaderTransformation: PASS_THROUGH
wasm:
filters:
- config:
'@type': type.googleapis.com/google.protobuf.StringValue
value: x-envoy,server
image: webassemblyhub.io/jameshbarton/remove-headers:v1.0
name: remove-headers
rootId: remove_headers
Note that you can test this for yourself without going through the entire WASM build, test, and deploy process as we have in this post. You can simply run the wasme deploy command as above using this image: webassemblyhub.io/jameshbarton/remove-headers:v1.0.
Test the WASM Filter
As you can see from the test below, our new configuration removes both the server and x-envoy-* headers from the response as expected.
% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
date: Wed, 07 Apr 2021 14:21:02 GMT
content-type: application/json
content-length: 234
access-control-allow-origin: *
access-control-allow-credentials: true
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "34.75.19.128",
"User-Agent": "curl/7.64.1",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000"
},
"origin": "10.104.2.17",
"url": "http://34.75.19.128/get"
}
Local Testing Notes
In this section, we moved directly to testing on the remote Kubernetes cluster. In most cases though, you’ll need to spin through a few edit-test-debug cycles before you’re ready to deploy. If that’s the case for you, you’ll want to consider the more efficient option of local testing using a local Envoy instance rather than a remote cluster. This documentation provides the details.
If you do opt for local testing, pay close attention to the embedded Istio version contained in the local Envoy instance. At this writing, it is quite old by default (Istio 1.5). So you may need to override that default to use a more recent Istio image. The example below worked well for much of our local testing.
wasme deploy envoy webassemblyhub.io/jameshbarton/rm-headers:v0.1 --config "x-envoy" --envoy-image docker.io/istio/proxyv2:1.8.1
Note that this advice may not be valid in the future. Expect that WASM support tools will handle this much more gracefully soon.
Evaluate Performance
Finally, we will conduct the same performance evaluation with the WASM filter as we did with the native Envoy filter. And we will use the same environment with the same settings — 10,000 requests with the default maximum of 50 client threads — against a warmed-up but untuned GKE cluster with three n1-standard-2 VMs.
% hey -n 10000 http://34.75.19.128/get Summary: Total: 11.5742 secs Slowest: 0.1715 secs Fastest: 0.0331 secs Average: 0.0568 secs Requests/sec: 863.9870 Total data: 2750000 bytes Size/request: 275 bytes Response time histogram: 0.033 [1] | 0.047 [1486] |■■■■■■■■■ 0.061 [6370] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.075 [1428] |■■■■■■■■■ 0.088 [306] |■■ 0.102 [163] |■ 0.116 [95] |■ 0.130 [70] | 0.144 [54] | 0.158 [14] | 0.171 [13] | Latency distribution: 10% in 0.0455 secs 25% in 0.0490 secs 50% in 0.0533 secs 75% in 0.0594 secs 90% in 0.0698 secs 95% in 0.0823 secs 99% in 0.1265 secs Details (average, fastest, slowest): DNS+dialup: 0.0003 secs, 0.0331 secs, 0.1715 secs DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs req write: 0.0000 secs, 0.0000 secs, 0.0007 secs resp wait: 0.0565 secs, 0.0331 secs, 0.1676 secs resp read: 0.0000 secs, 0.0000 secs, 0.0135 secs Status code distribution: [200] 10000 responses
With the WASM filter, we achieved the same functional results as before with throughput of 863.99 requests per second, an average response time of 56.8 milliseconds and a p99 of 126.5 milliseconds. Comparing these results to the native Envoy filter, the WASM filter throughput lagged by just 2.2% (863.99 vs. 883.86 requests per second), the average response time was 2.3% slower (56.8 vs 55.5 milliseconds), and the p99 lagged by 3.9% (126.5 vs. 121.7 milliseconds.)
As we said before, this is not a rigorous benchmark test and of course, you may see different results. You should always benchmark with your own workloads in your own environment before making deployment decisions.
However, with those caveats, the custom WASM filter works quite well from a performance standpoint next to its native Envoy counterpart.
Watch the Demo
See the demo based on this blog post that was presented at KubeCon Europe 2021.
Learn More
In this blog post, we explored how to solve an information leakage problem using Gloo Edge with both a native Envoy filter and a custom WASM filter. We walked step-by-step through each approach and accomplished the desired results with minimal performance differences between the two approaches.
All of the code used in this guide is available on github. The final WASM filter image is available for deployment from WebAssembly Hub: webassemblyhub.io/jameshbarton/remove-headers:v1.0
- Explore the documentation for Gloo Edge and WebAssembly.
- Request a live demo or trial for Gloo Edge Enterprise.
- See video content on the solo.io YouTube channel.
- Questions? Join the Solo.io Slack community and check out the community #gloo-edge and #wasm channels.
Acknowledgments
A big thank you to Shane O’Donnell for working with me to understand the nuances of building and testing WASM filters with AssemblyScript.
BACK TO BLOG
About the Author
