Envoy Proxy — high severity vulnerabilities that can lead to exposure of unauthorized services.

Overview

Two vulnerabilities have been discovered in the Envoy proxy that can potentially allow unauthorized access to backend resources. They are classified as of high severity according to the CVSS methodology and immediate action is needed.

CVE-2019–9900 (CVSS score 8.3)

When parsing HTTP/1.x header values, Envoy 1.9 and before does not reject embedded zero characters (NUL, ASCII 0x0). This allows remote attackers crafting header values containing embedded NUL characters to potentially bypass header matching rules, gaining access to unauthorized resources.
GitHub issue — https://github.com/envoyproxy/envoy/issues/6434

Envoy was failing to comply with the HTTP 1.1 specification with regards to resetting or closing the connection on the presence of invalid headers, denoting a malformed request, as specified here.

Envoy relies on a C library called http-parser to process request and responses, which is where the vulnerability ultimately is present. The team responsible for the parser library has been notified but no fix was provided as of the publishing of this communication. This can potentially lead to impacts on other components that depend on that same library.

To provide the fix, the Envoy Product Security Team implemented the needed changes on Envoy.

CVE-2019–9901 (CVSS score 8.3)

Envoy does not normalize HTTP URL paths in Envoy 1.9 and before. A remote attacker may craft a path with a relative path, e.g. something/../admin, to bypass access control, e.g. a block on /admin. A backend server could then interpret the non-normalized path and provide an attacker access beyond the scope provided for by the access control policy.
GitHub issue — https://github.com/envoyproxy/envoy/issues/6435

Impact

These CVEs impact all Envoy 1.9 versions and prior as well as the respective Envoy distributions that rely on those versions. Solo.io Gloo v0.13.10 and prior, and Gloo Enterprise v0.13.8 and prior, are affected by these CVEs.

Detection & Diagnosis — Determine if your system is vulnerable

Solo.io has created a Labs application to assist in the detection of exposure to these vulnerabilities. Use the links below to download our tool and assess if your system is vulnerable. The tool was tested on multiple Envoy distributions and the source code is publicly available here. We recommend using the provided hashes to verify the authenticity of the binaries.

SHA-256
dd8074d7843f0e3a8a8370fc604a146327becc808e843c4842ac4f0ab62b1900 envoy-cves-linux
SHA-256
9ad9eb69a280cdd5f717b6ca34aa3ff6787a5a225324efa6fd3f035e0d26ecd0 envoy-cves-darwin

The tool does not aim to exploit the vulnerabilities and its main intention is to verify if your systems have been patched with the fix.

Below you see an example of the tool being applied against a vulnerable version of Envoy.

If your system contains the patch, the tool will inform that you’re not vulnerable to these CVEs.

How to use the detection tool?

  • If Envoy is in your path, simply run it:
$ envoy-cves
✔ Success! your envoy was tested and is immune to CVE-2019–9901. Make sure that the option normalize_path is turned on in your HCM settings.
✔ Success! your envoy was tested and is immune to CVE-2019–9900
  • If not, provide the path to Envoy in a flag:
$ envoy-cves — envoy=/path/to/envoy
✘ Fail! your envoy did not normalize the path — it is vulnerable to CVE-2019–9901
✘ Fail! your envoy accepts nil in headers — it is vulnerable to CVE-2019–9900
  • Running inside a container:

The envoy-cves tool is a single static binary and can easily be used in a docker container. For example, to test the tool with the official Envoy container image, run this:

$ wget https://github.com/solo-io/envoy-cves/releases/download/v0.1.0/envoy-cves-linux
$ chmod +x envoy-cves
$ docker run -v $PWD/envoy-cves:/bin/envoy-cves — entrypoint=/bin/envoy-cves envoyproxy/envoy
  • Running inside Kubernetes, attaching to an already running Envoy:

Copy and run inside kubernetes using kubectl.

$ kubectl -n NAMESPACE cp envoy-cves POD_WITH_ENVOY:/tmp/envoy-cves
$ kubectl -n NAMESPACE exec POD_WITH_ENVOY /tmp/envoy-cves
✔ Success! your envoy was tested and it is immune to CVE-2019–9901. Make sure the option normalize_path is turned on in your HCM settings.
✔ Success! your envoy was tested and it is immune to CVE-2019–9900

Resolve — Take Action

Solo.io recommends users of the Gloo project and Gloo Enterprise product to immediately update their versions:

Gloo Community— Update to v0.13.10-patch1

kubectl apply https://github.com/solo-io/gloo/releases/download/v0.13.10-patch1/gloo-gateway.yaml

If you installed Gloo on a namespace other than gloo-system, which is the default namespace, make sure you make the necessary changes.

Gloo Enterprise — download updated glooctl

Next step is to run glooctl install with the options that best if youIf you are running this command against an already installed Gloo Enterprise, the new definition with the patch will be applied and your impacted Gloo Enterprise gateway will be updated.

The patched version addresses both CVEs thus reducing your exposure.

Users that rely on the Community distribution of Envoy are also strongly encouraged to patch their systems to the patched version provided by the Community. If you’re willing to run Envoy and would like to take advantage of an easy to use control plane, we recommend Gloo.

Join us on our Slack if you need assistance.

Acknowledgments

Solo.io would like to thank Harvey Tuch and Erlend Oftedal for discovering the CVEs and the extended Envoy community for working together to provide a fix.