From Zero to Gloo Edge in 15 Minutes: Amazon EKS Edition

How long does it take to configure your a cloud-native application on an open-source API gateway using Envoy Proxy with EKS?

Not an Amazon EKS user? Not a problem. Check out one of these alternative approaches:

eksctl create cluster --name zero-to-gateway --region us-east-2

[ℹ]  eksctl version 0.34.0
[ℹ]  using region us-east-2
[ℹ]  setting availability zones to [us-east-2c us-east-2a us-east-2b]
[ℹ]  subnets for us-east-2c - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for us-east-2a - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for us-east-2b - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-02e2ba93" will use "ami-0170c40fb8fc775fd" [AmazonLinux2/1.18]
[ℹ]  using Kubernetes version 1.18
[ℹ]  creating EKS cluster "zero-to-gateway" in "us-east-2" region with un-managed nodes
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-2 --cluster=zero-to-gateway'
[ℹ]  CloudWatch logging will not be enabled for cluster "zero-to-gateway" in "us-east-2"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=us-east-2 --cluster=zero-to-gateway'
[ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "zero-to-gateway" in "us-east-2"
[ℹ]  2 sequential tasks: { create cluster control plane "zero-to-gateway", 3 sequential sub-tasks: { tag cluster, create addons, create nodegroup "ng-02e2ba93" } }
[ℹ]  building cluster stack "eksctl-zero-to-gateway-cluster"
[ℹ]  deploying stack "eksctl-zero-to-gateway-cluster"
[✔]  tagged EKS cluster (used-by=jimb)
[ℹ]  building nodegroup stack "eksctl-zero-to-gateway-nodegroup-ng-02e2ba93"
[ℹ]  --nodes-min=2 was set automatically for nodegroup ng-02e2ba93
[ℹ]  --nodes-max=2 was set automatically for nodegroup ng-02e2ba93
[ℹ]  deploying stack "eksctl-zero-to-gateway-nodegroup-ng-02e2ba93"
[ℹ]  waiting for the control plane availability...
[✔]  saved kubeconfig as "/Users/jibarton/.kube/config"
[ℹ]  no tasks
[✔]  all EKS cluster resources for "zero-to-gateway" have been created
[ℹ]  adding identity "arn:aws:iam::410461945957:role/eksctl-zero-to-gateway-nodegroup-NodeInstanceRole-15JOF7LDMA91L" to auth ConfigMap
[ℹ]  nodegroup "ng-02e2ba93" has 0 node(s)
[ℹ]  waiting for at least 2 node(s) to become ready in "ng-02e2ba93"
[ℹ]  nodegroup "ng-02e2ba93" has 2 node(s)
[ℹ]  node "ip-192-168-25-133.us-east-2.compute.internal" is ready
[ℹ]  node "ip-192-168-91-116.us-east-2.compute.internal" is ready
[ℹ]  kubectl command should work with "/Users/jibarton/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "zero-to-gateway" in "us-east-2" region is ready

kubectl config get-contexts

CURRENT   NAME                                              CLUSTER                                           AUTHINFO                                          NAMESPACE
*         jim@zero-to-gateway.us-east-2.eksctl.io   zero-to-gateway.us-east-2.eksctl.io               jim@zero-to-gateway.us-east-2.eksctl.io
          kind-demo                                 kind-demo                                         kind-demo                                        kind-demo

kubectl config use-context jim@zero-to-gateway.us-east-2.eksctl.io

Switched to context "jim@zero-to-gateway.us-east-2.eksctl.io".

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-svc-dpl.yaml

serviceaccount/httpbin created
service/httpbin created
deployment.apps/httpbin created

kubectl get pods -l app=httpbin

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-66cdbdb6c5-2cnm7   1/1     Running   0          21m

curl -sL https://run.solo.io/gloo/install | sh
export PATH=$HOME/.gloo/bin:$PATH

glooctl version

Client: {"version":"1.7.10"}
Server: version undefined, could not find any version of gloo running

glooctl install gateway

Creating namespace gloo-system... Done.
Starting Gloo Edge installation...
Gloo Edge was successfully installed!

It should take less than a minute for the full Gloo Edge system to be ready for use. You can use this bash script to notify you when everything is ready to go:

until kubectl get ns gloo-system
do
sleep 1
done

until [ $(kubectl -n gloo-system get pods -o jsonpath='{range .items[*].status.containerStatuses[*]}{.ready}{"\n"}{end}' | grep false -c) -eq 0 ]; do
echo "Waiting for all the gloo-system pods to become ready"
sleep 1
done

echo "Gloo Edge deployment is ready :-)"

The system will respond:

NAME          STATUS   AGE
gloo-system   Active   15s
Waiting for all the gloo-system pods to become ready
Waiting for all the gloo-system pods to become ready
Waiting for all the gloo-system pods to become ready
Gloo Edge deployment is ready :-)

That’s all you need to do to install Gloo Edge. Notice that we did not install or configure any kind of external database to manage Gloo artifacts. That’s because the product was architected from Day 1 to be Kubernetes-native. All artifacts are expressed as Kubernetes Custom Resources (CRDs), and they are all stored in native etcd storage. Consequently, Gloo Edge leads to more resilient and less complex systems than alternatives that are either shoe-horned into Kubernetes or require external moving parts.

Note that everything we do in this getting-started exercise runs on the open-source version of Gloo Edge. There is also an enterprise edition of Gloo Edge that adds many features to support advanced authentication and authorization, rate limiting, and observability, to name a few.  You can see a feature comparison here. If you’d like to work through this blog post using Gloo Edge Enterprise instead, then please request a free trial here.

Discover K8s services with Envoy Proxy with EKS

A unique feature of Gloo Edge is its ability to discover Kubernetes (K8s) services and wrap them into an Upstream abstraction. Upstreams represent targets to which request traffic can be routed.  To learn more about how Upstreams operate in a Gloo Edge environment, see the product documentation here and here, and the API reference here.

Explore Service Discovery

Let’s use the glooctl utility to explore the catalog of Upstreams that Gloo Edge has already compiled within our kind cluster. You can run:

glooctl get upstreams

And you’ll see:

+-------------------------------+------------+----------+------------------------------+
|           UPSTREAM            |    TYPE    |  STATUS  |           DETAILS            |
+-------------------------------+------------+----------+------------------------------+
| default-httpbin-8000          | Kubernetes | Accepted | svc name:      httpbin       |
|                               |            |          | svc namespace: default       |
|                               |            |          | port:          8000          |
|                               |            |          |                              |
| default-kubernetes-443        | Kubernetes | Accepted | svc name:      kubernetes    |
|                               |            |          | svc namespace: default       |
|                               |            |          | port:          443           |
|                               |            |          |                              |
... abridged ...
| kube-system-kube-dns-9153     | Kubernetes | Accepted | svc name:      kube-dns      |
|                               |            |          | svc namespace: kube-system   |
|                               |            |          | port:          9153          |
|                               |            |          |                              |
+-------------------------------+------------+----------+------------------------------+

Notice in particular the default-httpbin-8000 Upstream that corresponds to the httpbin service we installed earlier.

Explore Function Discovery with OpenAPI

We could begin routing to this newly discovered httpbin Upstream right away. Before we do that, let’s explore advanced function discovery features that ship with open-source Gloo Edge. Function discovery is supported for both OpenAPI / REST and gRPC interfaces. In this example, we will associate an OpenAPI document with the httpbin Upstream and then observe the discovery feature at work.

We need to enable function discovery on the default namespace where the service is deployed. Since not all users employ OpenAPI or gRPC interfaces, and because function discovery can become resource-intensive, it is disabled by default. We could have enabled it via helm values at installation time; instead, we will do it here with kubectl:

kubectl label namespace default discovery.solo.io/function_discovery=enabled

Which confirms:

namespace/default labeled

Second, we will modify the httpbin Upstream to associate an OpenAPI document. There’s nothing unique or Gloo-specific in the OpenAPI document itself; it’s just an OpenAPI spec for a standard REST interface. You can see the full spec for httpbin here and interact with the individual operations in the httpbin sandbox here.

Let’s take a look at the modifications to the generated httpbin Upstream. All we’re doing is adding a URL to the Upstream that locates the OpenAPI specification for the httpbin service.

    serviceSpec:
      rest:
        swaggerInfo:
          url: https://raw.githubusercontent.com/jameshbarton/solo-resources/main/zero-to-gateway/httpbin-openapi.json

Now we’ll apply this change to the Upstream:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-openapi-us.yaml

And it confirms:

upstream.gloo.solo.io/default-httpbin-8000 configured

Now when we use glooctl to inspect the upstream and compare it with what we had before, you can see that Gloo Edge has discovered (with the guidance of the OpenAPI document) a number of individual operations being published from the httpbin service. This will allow us to be much more precise and avoid errors as we establish routing rules. Run:

glooctl get upstream default-httpbin-8000

You should see:

+----------------------+------------+----------+------------------------+
|       UPSTREAM       |    TYPE    |  STATUS  |        DETAILS         |
+----------------------+------------+----------+------------------------+
| default-httpbin-8000 | Kubernetes | Accepted | svc name:      httpbin |
|                      |            |          | svc namespace: default |
|                      |            |          | port:          8000    |
|                      |            |          | REST service:          |
|                      |            |          | functions:             |
|                      |            |          | - /anything            |
|                      |            |          | - /base64              |
|                      |            |          | - /brotli              |
|                      |            |          | - /bytes               |
|                      |            |          | - /cache               |
|                      |            |          | - /deflate             |
|                      |            |          | - /delay               |
|                      |            |          | - /delete              |
|                      |            |          | - /get                 |
|                      |            |          | - /gzip                |
|                      |            |          | - /headers             |
|                      |            |          | - /ip                  |
|                      |            |          | - /patch               |
|                      |            |          | - /post                |
|                      |            |          | - /put                 |
|                      |            |          | - /redirect-to         |
|                      |            |          | - /response-headers    |
|                      |            |          | - /status              |
|                      |            |          | - /stream              |
|                      |            |          | - /user-agent          |
|                      |            |          | - /uuid                |
|                      |            |          | - /xml                 |
|                      |            |          |                        |
+----------------------+------------+----------+------------------------+

Controlling Envoy Proxy with EKS

In this section, we’ll establish and test routing rules that are the core of the proxy configuration, and also show how to establish timeout policies from the proxy.

Configure Simple Routing with CLI

Let’s begin our routing configuration with the simplest possible route to expose the /get operation on httpbin. This endpoint simply replies with its response the headers and any other arguments passed into the service.

We’ll use the glooctl utility to get started:

glooctl add route \
  --path-exact /api/httpbin/get \
  --dest-name default-httpbin-8000 \
  --prefix-rewrite /get

And can see the output:

{"level":"info","ts":"2021-06-023T14:22:58.502-0400","caller":"add/route.go:156","msg":"Created new default virtual service","virtualService":"virtual_host:{domains:\"*\"  routes:{matchers:{exact:\"/api/httpbin/get\"}  route_action:{single:{upstream:{name:\"default-httpbin-8000\"  namespace:\"gloo-system\"}}}  options:{prefix_rewrite:{value:\"/get\"}}}}  status:{}  metadata:{name:\"default\"  namespace:\"gloo-system\"  resource_version:\"437677\"  generation:1}"}
+-----------------+--------------+---------+------+---------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  | STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+---------+-----------------+----------------------------------+
| default         |              | *       | none | Pending |                 | /api/httpbin/get ->              |
|                 |              |         |      |         |                 | gloo-system.default-httpbin-8000 |
|                 |              |         |      |         |                 | (upstream)                       |
+-----------------+--------------+---------+------+---------+-----------------+----------------------------------+

This glooctl invocation created a Gloo Edge VirtualService component, which is named default by default. It routes any request to the path /api/httpbin/get to the httpbin /get endpoint. Attempting to reach any other endpoint on the httpbin service will be rejected.

Note that when the route is initially created, the status of the route is Pending. Issue the glooctl get virtualservice default command and observe that the status has now changed from Pending to Accepted.

+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| default         |              | *       | none | Accepted |                 | /api/httpbin/get ->              |
|                 |              |         |      |          |                 | gloo-system.default-httpbin-8000 |
|                 |              |         |      |          |                 | (upstream)                       |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+

Confirm a Load Balancer is established

Now that a VirtualService is in place, Gloo Edge establishes an AWS load balancer that can be accessed via its hostname to invoke service endpoints published via the gateway.  You can see this hostname when you look at this entry for the gateway-proxy (Envoy proxy) service:

kubectl get service gateway-proxy -n gloo-system

Here is the response with the load balancer’s hostname reported as EXTERNAL-IP:

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)                      AGE
gateway-proxy   LoadBalancer   10.100.10.46   a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com   80:30593/TCP,443:32752/TCP   6h37m

You can also see the newly configured load balancer in the AWS console:

By default, Gloo Edge establishes an AWS Classic Load Balancer (CLB) to expose our gateway to the outside world. However, both Natural (NLB) and Application (ALB) Load Balancers are supported as well. Many Gloo Edge users have had success using the TCP Layer 4 NLBs in conjunction with the Gloo Edge Layer 7 gateway. For further information on AWS load balancer integration, see the product documentation.

Test the Simple Route with Curl

We will test our new Route via our AWS load balancer in this section. Let’s use the glooctl CLI to inspect the URL exposed from the proxy and assign that to a shell variable, like this:

proxy_url=$(glooctl proxy url)
echo $proxy_url

And it responds with the http prefix of the exposed AWS load balancer URL:

http://a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com:80

We will now use this glooctl utility with curl to invoke httpbin endpoints. In this case, we will add the curl -i option to show the HTTP response code and headers:

curl ${proxy_url}/api/httpbin/get -i

This command should complete successfully.

HTTP/1.1 200 OK
server: envoy
date: Wed, 23 Jun 2021 20:30:41 GMT
content-type: application/json
content-length: 404
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 2

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "5000",
    "X-Envoy-Original-Path": "/api/httpbin/get"
  },
  "origin": "192.168.77.109",
  "url": "http://a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com/get"
}

Note that if we attempt to invoke another valid endpoint /delay on the httpbin service, it will fail with a 404 Not Found error. Why? This is because our VirtualService routing policy is only exposing access to /get, one of the many endpoints available on the service. If we enter:

curl ${proxy_url}/api/httpbin/delay/1 -i

You’ll see:

HTTP/1.1 404 Not Found
date: Fri, 23 Jun 2021 20:32:01 GMT
server: envoy
content-length: 0

Explore Complex Routing with Regex Patterns

Let’s assume that now we DO want to expose other httpbin endpoints like /delay. Our initial VirtualService is inadequate, because it is looking for an exact path match with /api/httpbin/get. Here is the core YAML for that VirtualService as constructed by the glooctl add route command we issued earlier.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - exact: /api/httpbin/get
      options:
        prefixRewrite: /get
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

This time, rather than using the glooctl CLI, let’s manipulate the VirtualService directly. We’ll modify the matchers: stanza to match the path prefix /api/httpbin and replace it with /. So a path like /api/httpbin/delay/1 will be sent to httpbin with the path /delay/1. Now it will look like this:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /api/httpbin
      options:
        regexRewrite:
          pattern:
            regex: '/api/httpbin/'
          substitution: '/'
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

Let’s apply the modified VirtualService and test. Note that throughout this exercise, we are managing Gloo Edge artifacts using Kubernetes utilities like kubectl. That’s an important point because it allows developers to work with familiar tools when working with Gloo Edge configuration. It also benefits organization using GitOps strategies to manage deployments, as tools like ArgoCD and Flux are able to easily handle Gloo artifacts on Kubernetes.

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-regex.yaml

Note that you can safely ignore the “kubectl apply” warning below. As long as kubectl responds that the default VirtualService was configured, then your change was applied.

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
virtualservice.gateway.solo.io/default configured

Test Routing with Regex Patterns

When we used only a single route with an exact match pattern, we could only exercise the httpbin /get endpoint. Let’s now use curl to confirm that both /get and /delay work as expected.

% curl ${proxy_url}/api/httpbin/get -i
HTTP/1.1 200 OK
server: envoy
date: Wed, 23 Jun 2021 20:33:22 GMT
content-type: application/json
content-length: 404
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "5000",
    "X-Envoy-Original-Path": "/api/httpbin/get"
  },
  "origin": "192.168.77.109",
  "url": "http://a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com/get"
}
% curl ${proxy_url}/api/httpbin/delay/1 -i
HTTP/1.1 200 OK
server: envoy
date: Wed, 23 Jun 2021 20:33:25 GMT
content-type: application/json
content-length: 458
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1002

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "5000",
    "X-Envoy-Original-Path": "/api/httpbin/delay/1"
  },
  "origin": "192.168.77.109",
  "url": "http://a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com/delay/1"
}

Perfect! It works just as expected! For extra credit, try out some of the other endpoints published via httpbin as well.

Configure Timeouts

Don’t you hate it when you visit a website and the request just gets “lost”? You wait and wait. Maybe you see the network connection established, but then you wait some more, and still the request never completes.

Gloo Edge provides an easy-to-configure set of timeouts that you can apply to spare your valuable users this frustration. Like other Gloo features, it can be added to your policy with standard Kubernetes tooling, and without touching the source application. All we need to do add a timeout directive to our VirtualService. In this case, we will apply the timeout in the simplest fashion at the httpbin route level by adding timeout to our route options.

    routes:
    - matchers:
      - prefix: /api/httpbin
      options:
        timeout: '5s'  # Adding 5-second timeout HERE
        regexRewrite: 
          pattern:
            regex: '/api/httpbin/'
          substitution: '/'

Let’s apply this VirtualService change using kubectl:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-timeout.yaml

It confirms:

virtualservice.gateway.solo.io/default configured

Test Timeouts with httpbin delays

We will confirm that our new timeout policy works by using the httpbin /delay endpoint. We’ll specify a one second delay, and we expect everything to work just fine. Second, we’ll specify a longer delay, say eight seconds, and we will expect our timeout policy to be triggered and return a 504 Gateway Timeout error. Run:

curl ${proxy_url}/api/httpbin/delay/1 -i

This returns:

HTTP/1.1 200 OK
server: envoy
date: Wed, 23 Jun 2021 20:37:14 GMT
content-type: application/json
content-length: 458
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1003

{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Host": "a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com",
"User-Agent": "curl/7.64.1",
"X-Envoy-Expected-Rq-Timeout-Ms": "5000",
"X-Envoy-Original-Path": "/api/httpbin/delay/1"
},
"origin": "192.168.77.109",
"url": "http://a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com/delay/1"

Note that the operation completed successfully and that the one second delay was applied. The response header x-envoy-upstream-service-time: 1003 indicates that the request spent 1,003 milliseconds being processed by Envoy.

Now let’s switch to an eight second delay and see if our five second timeout triggers as expected. Execute:

% curl ${proxy_url}/api/httpbin/delay/8 -i
HTTP/1.1 504 Gateway Timeout
content-length: 24
content-type: text/plain
date: Fri, 23 Jun 2021 20:39:03 GMT
server: envoy

upstream request timeout

BOOM! Our simple timeout policy works just as expected, triggering a 504 Gateway Timeout error when the five second threshold is exceeded by our eight second httpbin delay.

Debugging Envoy Proxy with EKS

Sometimes debugging bad software configurations can be a pain. Gloo Edge engineers have made this as easy as possible, with documentation like this. However, as we have all experienced, it can be a challenge with any complex system. Here we’ll explore how to use the glooctl utility to assist in some simple debugging tasks.

Solve a Problem with glooctl CLI

A common source of Gloo Edge configuration errors is mistyping an upstream reference, perhaps when copy/pasting it from another source but “missing a spot” when changing the name of the Upstream target. In this example, we’ll simulate making an error like that, and then demonstrate how glooctl can be used to detect it.

Let’s apply a change to simulate the mistyping of an upstream config so that it is targeting a non-existent default-httpbin-8080 Upstream, rather than the correct default-httpbin-8000:

kubectl delete virtualservice default -n gloo-system
sleep 5
kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-bad-port.yaml

You should see:

virtualservice.gateway.solo.io "default" deleted
virtualservice.gateway.solo.io/default created

Note that we applied a sleep between deleting the VirtualService and re-creating it to ensure that we clear the working route from Envoy’s route cache before we create the new mis-configured route.

Now if we try to access one of our httpbin endpoints:

curl ${proxy_url}/api/httpbin/get -i

It likely fails with an error like this:

curl: (7) Failed to connect to 34.75.42.228 port 80: Connection refused

So we’ll deploy one of the Gloo Edge debugging features, the glooctl check utility, to help you manage Envoy Proxy with EKS. It performs checks on a number of Gloo resources, confirming that they are configured correctly and are interconnected with other resources correctly. For example, in this case, glooctl will detect the error in the mis-connection between a VirtualService and its Upstream target:

glooctl check

You can see the checks respond:

Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... 2 Errors!
Checking gateways... OK
Checking proxies... 1 Errors!
Error: 3 errors occurred:
	* Found virtual service with warnings: gloo-system default (Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.default-httpbin-8080 } not found)
	* Virtual service references unknown upstream: (Virtual service: gloo-system default | Upstream: gloo-system default-httpbin-8080)
	* Found proxy with warnings: gloo-system gateway-proxy
Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.default-httpbin-8080 } not found

The detected errors clearly identify that the VirtualService is pointed at an invalid destination.

So let’s reapply the previous configuration, and then we’ll confirm that the configuration is again clean:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-timeout.yaml

Now we get confirmation:

virtualservice.gateway.solo.io/default configured

Re-run glooctl check and observe that there are no problems. Our curl commands to the httpbin endpoint will also work again as expected:

Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... OK
Checking gateways... OK
Checking proxies... OK
No problems detected.

Observing Envoy Proxy with EKS

Finally, let’s tackle an exercise where we’ll learn about some simple observability tools that ship with open-source Gloo Edge.

Configure Simple Access Logging

The default Envoy Proxy logging configurations are very quiet by design. When working in extremely high-volume environments, verbose logs can potentially impact performance and consume excessive storage.

However, access logs are quite important at development and test time, and potentially in production as well. So let’s explore how to set up and consume simple access logs.

So far we have discussed Gloo Edge custom resources like Upstreams and VirtualServices. Upstreams represent the target systems to which Gloo routes traffic. VirtualServices represent the policies that determine how external requests are routed to those targets.

With access logging, we will consider another Gloo Edge component called a gateway. A gateway is a custom resource that configures the protocols and ports on which Gloo Edge listens for traffic. For example, by default Gloo Edge will have a gateway configured for HTTP and HTTPS traffic. More information on gateways is available here. Gloo Edge allows you to customize the behavior of your gateways in multiple ways. One of those ways is by adding access logs.

Let’s add the simplest, default access log configuration. We’ll simply add the following options to activate access logging. You can see the full gateway YAML that we’ll apply here.

  options:
    accessLoggingService:
      accessLog:
      - fileSink:
          path: /dev/stdout
          stringFormat: ""

Now let’s apply this change to enable access logging from our Envoy data plane:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/gateway-basic-access-logs.yaml

As before, you can safely ignore the benign kubectl warning:

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
gateway.gateway.solo.io/gateway-proxy configured

Now let’s generate some traffic to produce some access logs:

curl ${proxy_url}/api/httpbin/get
curl ${proxy_url}/api/httpbin/delay/1
curl ${proxy_url}/api/httpbin/delay/8

You should be able to view the resulting access logs by using the kubectl logs commands against the Envoy data plane pod:

kubectl logs -n gloo-system deploy/gateway-proxy

Here’s what we saw:

[2021-06-23T20:41:25.459Z] "GET /api/httpbin/get HTTP/1.1" 200 - 0 404 2 2 "-" "curl/7.64.1" "8bdb62de-d066-4d5b-a848-bdfda1cdcd8f" "a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com" "192.168.29.133:80"
[2021-06-23T20:41:25.558Z] "GET /api/httpbin/delay/1 HTTP/1.1" 200 - 0 458 1002 1002 "-" "curl/7.64.1" "28be8891-c6b6-48e7-b430-90ba1e0bfed5" "a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com" "192.168.29.133:80"
[2021-06-23T20:41:26.687Z] "GET /api/httpbin/delay/8 HTTP/1.1" 504 UT 0 24 5002 - "-" "curl/7.64.1" "aa441de0-47f4-49a0-92e9-195bf2d299d8" "a207de2527e7742b5a2096dd2bae1b55-2061871899.us-east-2.elb.amazonaws.com" "192.168.29.133:80"

Notice from the output the default string formatted access log for each of the operations we executed. You can see the paths of the operations, plus the HTTP response codes:  200 for the first two, and 504 (Gateway Timeout). Plus there is a host of other information.

While we are viewing these access logs using kubectl, you may want to export it for use with an enterprise log aggregator like ELK, Splunk, or Datadog. For example, Gloo Edge provides guidance for integrating with popular platforms like Datadog.

Customize Access Logging

Gloo Edge gateways can also be configured to produce access logs in other formats like JSON, and to customize the actual content that is published to those logs. We will do both of those things by replacing our previous access log configuration in the gateway component with this:

  options:
    accessLoggingService:
      accessLog:
      - fileSink:
          jsonFormat:
            # HTTP method name
            httpMethod: '%REQ(:METHOD)%'
            # Protocol. Currently either HTTP/1.1 or HTTP/2.
            protocol: '%PROTOCOL%'
            # HTTP response code. Note that a response code of ‘0’ means that the server never sent the
            # beginning of a response. This generally means that the (downstream) client disconnected.
            responseCode: '%RESPONSE_CODE%'
            # Total duration in milliseconds of the request from the start time to the last byte out
            clientDuration: '%DURATION%'
            # Total duration in milliseconds of the request from the start time to the first byte read from the upstream host
            targetDuration: '%RESPONSE_DURATION%'
            # Value of the "x-envoy-original-path" header (falls back to "path" header if not present)
            path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
            # Upstream cluster to which the upstream host belongs to
            upstreamName: '%UPSTREAM_CLUSTER%'
            # Request start time including milliseconds.
            systemTime: '%START_TIME%'
            # Unique tracking ID
            requestId: '%REQ(X-REQUEST-ID)%'
          path: /dev/stdout

More information on customizing access log content is provided here.

Now we will apply this gateway change, generate some additional traffic to our proxy, and view the resulting logs.

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/gateway-json-access-logs.yaml

This responds:

gateway.gateway.solo.io/gateway-proxy configured

We’ll use the same curl commands as before to generate some traffic:

curl ${proxy_url}/api/httpbin/get
curl ${proxy_url}/api/httpbin/delay/1
curl ${proxy_url}/api/httpbin/delay/8

Note that it may take a few seconds until the access log content is flushed to the logs:

kubectl logs -n gloo-system deploy/gateway-proxy | grep ^{ | jq

In the end, you should be able to see our customized JSON content, looking something like this:

{
  "upstreamName": "default-httpbin-8000_gloo-system",
  "clientDuration": 2,
  "path": "/api/httpbin/get",
  "httpMethod": "GET",
  "requestId": "0b005aa4-b7c9-42da-a40e-f5442ccb3388",
  "targetDuration": 2,
  "protocol": "HTTP/1.1",
  "systemTime": "2021-06-23T20:43:07.436Z",
  "responseCode": 200
}
{
  "targetDuration": 1002,
  "path": "/api/httpbin/delay/1",
  "protocol": "HTTP/1.1",
  "requestId": "2d812b40-add7-4b83-a2f5-2dbe18da88d3",
  "clientDuration": 1002,
  "upstreamName": "default-httpbin-8000_gloo-system",
  "systemTime": "2021-06-23T20:43:07.528Z",
  "httpMethod": "GET",
  "responseCode": 200
}
{
  "requestId": "a123ac4e-5bf6-4aa6-84fe-3a81efa2ab90",
  "protocol": "HTTP/1.1",
  "upstreamName": "default-httpbin-8000_gloo-system",
  "systemTime": "2021-06-23T20:43:08.657Z",
  "targetDuration": null,
  "path": "/api/httpbin/delay/8",
  "responseCode": 504,
  "clientDuration": 5004,
  "httpMethod": "GET"
}

Explore Envoy Proxy Metrics

Envoy publishes many metrics that may be useful for observing system behavior. In our very modest kind cluster for this exercise, you can count over 3,000 individual metrics! You can learn more about them in the Envoy documentation here.

Let’s take a quick look at a couple of the useful metrics that Envoy produces for every one of our Upstream targets.

We’ll port-forward the Envoy administrative port 19000 to our local workstation:

kubectl port-forward -n gloo-system deploy/gateway-proxy 19000 &

This shows:

Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000

Then let’s view two of the metrics that are most relevant to this exercise: one that counts the number of successful (HTTP 200) requests processed by our httpbin Upstream, and another that counts the number of gateway timeout (HTTP 504) requests against that same upstream:

curl -s http://localhost:19000/stats | grep -E 'cluster.default-httpbin-8000_gloo-system.upstream_rq_(200|504)'

Which gives us:

cluster.default-httpbin-8000_gloo-system.upstream_rq_200: 7
cluster.default-httpbin-8000_gloo-system.upstream_rq_504: 2

As you can see, on my instance we’ve processed seven good requests and two bad ones. If we apply the same three curl requests as before, we’d expect the number of 200 requests to increment by two, and the number of 504 requests to increment by one:

curl ${proxy_url}/api/httpbin/get
curl ${proxy_url}/api/httpbin/delay/1
curl ${proxy_url}/api/httpbin/delay/8

curl -s http://localhost:19000/stats | grep -E 'cluster.default-httpbin-8000_gloo-system.upstream_rq_(200|504)'

And that is exactly what we see!

cluster.default-httpbin-8000_gloo-system.upstream_rq_200: 9
cluster.default-httpbin-8000_gloo-system.upstream_rq_504: 3

If you’d like to have more tooling and enhanced visibility around system observability, we recommend taking a look at an Enterprise subscription to Gloo Edge so you can run Envoy Proxy with EKS in your own AWS environment. You can sign up for a free trial here.

Gloo Edge Enterprise provides out-of-the-box integration with both Prometheus and Grafana, allowing you to replace curl and grep with per-Upstream generated dashboards like this.

You can learn more about creating your own Grafana dashboards from Gloo Edge Enterprise metrics in this blog post.

Cleanup

If you’d like to cleanup the work you’ve done, you can delete your EKS cluster from the Google Cloud console or by using the eksctl CLI:

eksctl delete cluster --name zero-to-gateway --region us-east-2

You should see output similar to this if successful:

[ℹ]  eksctl version 0.34.0
[ℹ]  using region us-east-2
[ℹ]  deleting EKS cluster "zero-to-gateway"
[ℹ]  deleted 0 Fargate profile(s)
[✔]  kubeconfig has been updated
[ℹ]  cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
[ℹ]  2 sequential tasks: { delete nodegroup "ng-02e2ba93", delete cluster control plane "zero-to-gateway" [async] }
[ℹ]  will delete stack "eksctl-zero-to-gateway-nodegroup-ng-02e2ba93"
[ℹ]  waiting for stack "eksctl-zero-to-gateway-nodegroup-ng-02e2ba93" to get deleted
[ℹ]  will delete stack "eksctl-zero-to-gateway-cluster"
[✔]  all cluster resources were deleted

Learn More