[Tutorial] Secure the Edge with Gloo and get your A+ grade!

With the rise of “HTTPS first”, companies had to make the move to automated pipelines for certificate management. One of the most popular Certificate Authorities is Let’s Encrypt, certainly because of its free & self-service infrastructure for signing certificate requests.

In this blog post, we will explore Gloo Edge options around TLS. In order to facilitate the integration with Certificate Authorities, this tutorial uses Cert-Manager, backed by the Let’s Encrypt PKIs.

Technically, you will get a server certificate signed by the Let’s Encrypt CA, enable HSTS, and also restrict the list of allowed TLS versions and ciphers.

Prerequisites

First, you need a Kubernetes cluster where you will install Gloo Edge. An easy way to go is with the glooctl CLI:

curl -sL https://run.solo.io/gloo/install | sh
export PATH=$HOME/.gloo/bin:$PATH

Then, running the following command will install Gloo Edge:

glooctl install gateway

Gloo Edge should now be installed in your cluster:

Creating namespace gloo-system... Done.
Starting Gloo Edge installation...

Gloo Edge was successfully installed!

Then, you can deploy a basic backend service, like httpbin :

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yamlCopy

Gloo Edge discovers Kubernetes services automatically and creates a routable target called an Upstream. So, running the glooctl get upstreams command, you should be able to see a new Gloo Edge Upstream default-httpbin-8000, based on the naming convention namespace-serviceName-portNumber:

% glooctl get upstreams default-httpbin-8000
+----------------------+------------+----------+------------------------+
|       UPSTREAM       |    TYPE    |  STATUS  |        DETAILS         |
+----------------------+------------+----------+------------------------+
| default-httpbin-8000 | Kubernetes | Accepted | svc name:      httpbin |
|                      |            |          | svc namespace: default |
|                      |            |          | port:          8000    |
|                      |            |          |                        |
+----------------------+------------+----------+------------------------+

Use kubectl to create the following Gloo Edge VirtualService that will route all requests from your domain, here securetheedge.solo.io, to the new Upstream.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - securetheedge.solo.io
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

Run the following glooctl command to confirm that the new Route has been accepted by Gloo Edge.

% glooctl get virtualservice httpbin
+------------------+--------------+--------------+------+----------+-----------------+----------------------------------+
| VIRTUAL SERVICE  | DISPLAY NAME |   DOMAINS    | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES              |
+------------------+--------------+--------------+------+----------+-----------------+----------------------------------+
| httpbin          |              | <domain>     | none | Accepted |                 | / ->                             |
|                  |              |              |      |          |                 | gloo-system.default-httpbin-8000 |
|                  |              |              |      |          |                 | (upstream)                       |
+------------------+--------------+--------------+------+----------+-----------------+----------------------------------+

Gloo Edge integration with Cert-Manager

Configuring cert-manager

First, let’s install cert-manager via its Helm chart:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.2.0 \
  --set installCRDs=true

TIP: Cert-manager comes with a nice kubectl plugin, which is very handy when you want to check the status of your CSR.
Example: kubectl cert-manager status certificate my-cert

Now, let’s set up a ClusterIssuer CR (Custom Resource), which will act as a bridge between Certificates (one of Let’s Encrypt CRDs) and the Let’s Encrypt Certificate Authority:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <email address>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: baptiste-clusterissuer-prod-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          serviceType: ClusterIP # the service is exposed via Gloo Edge
          podTemplate:
            metadata:
              labels:
                role: acme-solver
      selector:
        dnsNames:
        - $(glooctl proxy address | cut -f 1 -d ':').nip.io
        - securetheedge.solo.io
EOF

There are two interesting points in the example above:

• First, once Let’s Encrypt tries to validate the ACME challenge, it will connect to your cluster and try to reach the solver service. This solver service is not exposed by any Ingress, since you’ve already deployed Gloo Edge, which can do the request routing. So, you can configure this ClusterIssuer CR, so that the solver pod is not exposed outside of the cluster, simply by setting this serviceType: ClusterIP parameter.
• Also, you can explicitly label the pod, thanks to the podTemplate block. This way, you will be able to target this pod, from a Kubernetes Service (see below). After that, if you have Gloo’s Discovery option enabled, then Gloo will automatically create an Upstream object, which will reference your Kubernetes Service. So, in fine, you will have a stable pointer to the solver pod (which is ephemeral).

Now, let’s create the kub Service:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: acme-solver
  namespace: gloo-system
spec:
  ports:
  - port: 8089
    protocol: TCP
    targetPort: 8089
  selector:
    role: acme-solver
EOF

Optional: if you disabled service discovery (UDS), you can manually create an Upstream with this command:

glooctl create upstream kube --name acme-solver-us --namespace gloo-system --kube-service acme-solver --kube-service-namespace gloo-system --kube-service-port 8089

Finally, let’s add a route to this solver, in your VirtualService:

# update the VS with our Upstream
kubectl apply -f - <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /.well-known/acme-challenge/
      routeAction:
        single:
          upstream:
            name: acme-solver-us
            namespace: gloo-system
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system
EOF

Alright! the cert-manager deployment is now all set!

Adding a new certificate to the server

It’s now time to request a new Certificate:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nip-io-cert-prod
  namespace: gloo-system
spec:
  secretName: nip-io-tls
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  commonName: securetheedge.solo.io
  dnsNames:
  - securetheedge.solo.io
EOF

Let’s check its status:

% kubectl cert-manager status certificate nip-io-cert-prod -n gloo-system
Name: nip-io-cert-prod
Namespace: gloo-system
Created at: 2021-03-30T10:05:19+02:00
Conditions:
  Ready: True, Reason: Ready, Message: Certificate is up to date and has not expired
DNS Names:
- securetheedge.solo.io
Events:  
Issuer:
  Name: letsencrypt
  Kind: ClusterIssuer
  Conditions:
    Ready: True, Reason: ACMEAccountRegistered, Message: The ACME account was registered with the ACME server
  Events:  
Secret:
  Name: nip-io-tls
  Issuer Country: US
  Issuer Organisation: Let's Encrypt
  Issuer Common Name: R3
  Key Usage: Digital Signature, Key Encipherment
  Extended Key Usages: Server Authentication, Client Authentication
  Public Key Algorithm: RSA
  Signature Algorithm: SHA256-RSA
  Subject Key ID: 7309e481573bb3db6518cb4aa183bdb7a5a56187
  Authority Key ID: 142eb317b75856cbae500940e61faf9d8b14c2c6
  Serial Number: 03d423b0f121df700f8be1b22750b91d31a0
  Events:  
Not Before: 2021-03-30T10:38:56+02:00
Not After: 2021-06-28T10:38:56+02:00
Renewal Time: 2021-05-29T10:38:56+02:00
No CertificateRequest found for this Certificate

Perfect! ?

Now, you can safely add this certificate to your VirtualService:

glooctl edit vs --name httpbin --namespace gloo-system --ssl-secret-name nip-io-tls --ssl-secret-namespace gloo-system

And voilà! this will get you a ‘B‘ grade on SSLLabs. But, you deserve better than that!

Upgrading TLS

Running a Server test on SSLLabs will report a bunch of deprecated ciphers and TLS protocol versions (or even worse, “SSL“). You want to get rid of SSL 3.0 and also of TLS 1.0 and TLS 1.1

Additionally, it’s a good practice to shorten the list of ciphers (encryption suites) that you support, because some of them are marked as weak.

Easy-peasy! just tell it to Gloo (which will tell Envoy):

% kubectl -n gloo-system patch --type merge vs/httpbin -p "
spec:
  sslConfig:
    parameters:
      minimumProtocolVersion: TLSv1_2
      cipherSuites:
      - ECDHE-RSA-AES128-GCM-SHA256
      - ECDHE-RSA-AES256-GCM-SHA384
      - ECDHE-RSA-CHACHA20-POLY1305
"

Tadaa! you got your ‘A‘ grade!

Enabling HSTS

HSTS stands for HTTP Strict Transport Security. It’s a means to tell browsers they should now connect to your website over HTTPS, even if a user clicks an HTTP link. But for this to work, end-users must first visit your website once (using HTTPS). A few rationales are published here: https://https.cio.gov/hsts/ and additional information can be found on Mozilla’s developer website (in particular about the preload option).

So, adding this response header is pretty straightforward, as you can see in this example:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  sslConfig:
    ...
  virtualHost:
    domains:
    - securetheedge.solo.io
    routes:
    ...
    options:
      headerManipulation:
        responseHeadersToAdd:
        - header:
            key: Strict-Transport-Security
            value: 'max-age=31536000; includeSubDomains; preload'

And here we are! you should now get your well-deserved ‘A+‘ grade!

Bonus

Controlling the CAs that can issue certificates for your domains and sub-domains is a good practice. In this regard, it’s recommended to add a special record to your DNS, named CAA.

Here is a permissive value for allowing the Let’s Encrypt CA:

% dig +short solo.io CAA
0 issue "letsencrypt.org"

Learn More

In this guide, we described how to upgrade the security level of your TLS termination, with Gloo Edge.

You can request a free trial of Gloo Edge Enterprise and Gloo Portal today here. To connect, join the #gloo-edge and #gloo-portal channels in the Solo.io Slack.