From Zero to Gloo Edge in 15 Minutes: GKE Edition

15 minutes blog header

How long does it take to configure your a cloud-native application on an open-source API Gateway?

How about 15 minutes? Give us that much time and we’ll give you a Kubernetes-hosted application accessible via a gateway configured with policies for routing, service discovery, timeouts, debugging, access logging, and observability. We’ll host all of this on Google Kubernetes Engine (GKE), one of the most popular public-cloud options for hosting Kubernetes workloads. The Gloo Edge API gateway builds on the foundation of Envoy Proxy, the open-source proxy in Istio, itself one of the most influential enterprise cloud projects available today.

Not an GKE user? Not a problem. Check out alternative approaches that use either a local KinD cluster, an AWS Elastic Kubernetes Service (EKS) cluster, or a Red Hat OpenShift cluster instead.

If you have questions, please reach out on the Solo #edge-quickstart Slack channel.

Ready? Set? Go!

Prerequisites

We’re going to do the work on your local workstation and a GKE cluster, plus the CLI utilities kubectl, curl, and jq. Make sure these are all available to you before jumping into the next section. I’m building this on MacOS but other platforms should be perfectly fine as well.

INSTALL

Let’s start by installing the platform and application components we need for this exercise.

Establish GKE Cluster

If you do not have access to a GKE cluster today, check with your local Google Cloud administrator to help you establish one. If that’s not an option for you, Google will happily sign you up for a free trial of its Cloud Platform products with $300 in credit to use over the next 90 days. That will allow you to complete this exercise with credit to spare.

Once you have Google Cloud access with sufficient permissions to access or create your own GKE cluster, then you’re ready to go. If you’re creating your own cluster, then you have two options:  the Google Cloud console or the gcloud CLI.

If you choose the console route, this interface provides you with wizards to walk you through cluster creation, once you have established your identity and identified a project to host your cluster.

If you already have Google Cloud access, then the gcloud CLI utility may provide a simpler alternative to the console. See this documentation if you would like to try that approach.

Try this command with the gcloud CLI to create a modest demo cluster for this exercise:

gcloud container clusters create my-edge-demo --num-nodes=3 --machine-type=n1-standard-2 -q

If you have adequate permissions on the project, then you should see output that looks something like this:

Creating cluster my-edge-demo in us-east1-d... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/solo-test-236622/zones/us-east1-d/clusters/my-edge-demo].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-east1-d/my-edge-demo?project=solo-test-236622
kubeconfig entry generated for my-edge-demo.
NAME           LOCATION    MASTER_VERSION   MASTER_IP    MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
my-edge-demo   us-east1-d  1.19.9-gke.1900  34.74.15.17  n1-standard-2  1.19.9-gke.1900  3          RUNNING

Running a kubectl config command should confirm that you’re ready to go:

kubectl config get-contexts

Note the new context containing the name of your freshly created GKE cluster (my-edge-demo in this case). It should also be selected as the current context that you’re using.

CURRENT   NAME                                              CLUSTER                                           AUTHINFO                                          NAMESPACE
*         gke_solo-test-236622_us-east1-d_my-edge-demo      gke_solo-test-236622_us-east1-d_my-edge-demo      gke_solo-test-236622_us-east1-d_my-edge-demo
          kind-demo                                         kind-demo                                         kind-demo

If it isn’t selected as the current context, you can remedy that with kubectl, using something like this:

kubectl config use-context gke_solo-test-236622_us-east1-d_my-edge-demo

The response should match:

Switched to context "gke_solo-test-236622_us-east1-d_my-edge-demo".

Install htttpbin Application

HTTPBIN is a great little REST service that can be used to test a variety of http operations and echo the response elements back to the consumer. We’ll use it throughout this exercise. We’ll install the httpbin service on our GKE cluster. Run this command:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-svc-dpl.yaml

You should see:

serviceaccount/httpbin created
service/httpbin created
deployment.apps/httpbin created

You can confirm that the httpbin pod is running by searching for pods with an app label of httpbin:

kubectl get pods -l app=httpbin

And you will see something like this:

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-66cdbdb6c5-2cnm7   1/1     Running   0          21m

 

Install glooctl Utility

GLOOCTL is a command-line utility that allows users to view, manage and debug Gloo Edge deployments, much like a Kubernetes user employs the kubectl utility. Let’s install glooctl on our local workstation:

curl -sL https://run.solo.io/gloo/install | sh
export PATH=$HOME/.gloo/bin:$PATH

We’ll test out the installation using the glooctl version command. It responds with the version of the CLI client that you have installed. However, the server version is undefined since we have not yet installed Gloo Edge. Enter:

glooctl version

Which responds:

Client: {"version":"1.7.10"}
Server: version undefined, could not find any version of gloo running

Install Gloo Edge

Finally, we will complete the INSTALL phase by configuring an instance of open-source Gloo Edge on our GKE cluster.

glooctl install gateway

And you’ll see:

Creating namespace gloo-system... Done.
Starting Gloo Edge installation...
Gloo Edge was successfully installed!

It should take less than a minute for the full Gloo Edge system to be ready for use. You can use this bash script to notify you when everything is ready to go:

until kubectl get ns gloo-system
do
sleep 1
done

until [ $(kubectl -n gloo-system get pods -o jsonpath='{range .items[*].status.containerStatuses[*]}{.ready}{"\n"}{end}' | grep false -c) -eq 0 ]; do
echo "Waiting for all the gloo-system pods to become ready"
sleep 1
done

echo "Gloo Edge deployment is ready :-)"

The system will respond:

NAME          STATUS   AGE
gloo-system   Active   15s
Waiting for all the gloo-system pods to become ready
Waiting for all the gloo-system pods to become ready
Waiting for all the gloo-system pods to become ready
Gloo Edge deployment is ready :-)

That’s all that’s required to install Gloo Edge. Notice that we did not install or configure any kind of external database to manage Gloo artifacts. That’s because the product was architected to be Kubernetes-native. All artifacts are expressed as Kubernetes Custom Resources, and they are all stored in native etcd storage. Consequently, Gloo Edge leads to more resilient and less complex systems than alternatives that are either shoe-horned into Kubernetes or require external moving parts.

Note that everything we do in this getting-started exercise runs on the open-source version of Gloo Edge. There is also an enterprise edition of Gloo Edge that adds features to support advanced authentication and authorization, rate limiting, and observability, to name a few. You can see a feature comparison here. If you’d like to work through this blog post using Gloo Edge Enterprise instead, then request a free trial here.

DISCOVER

A unique feature of Gloo Edge is its ability to discover Kubernetes services and wrap them into an Upstream abstraction. Upstreams represent targets to which request traffic can be routed.  To learn more about how Upstreams operate in a Gloo Edge environment, see the product documentation here and here, and the API reference here.

Explore Service Discovery

Let’s use the glooctl utility to explore the catalog of Upstreams that Gloo Edge has already compiled within our GKE cluster. You can run:

glooctl get upstreams

And you’ll see:

+-------------------------------+------------+----------+------------------------------+
|           UPSTREAM            |    TYPE    |  STATUS  |           DETAILS            |
+-------------------------------+------------+----------+------------------------------+
| default-httpbin-8000          | Kubernetes | Accepted | svc name:      httpbin       |
|                               |            |          | svc namespace: default       |
|                               |            |          | port:          8000          |
|                               |            |          |                              |
| default-kubernetes-443        | Kubernetes | Accepted | svc name:      kubernetes    |
|                               |            |          | svc namespace: default       |
|                               |            |          | port:          443           |
|                               |            |          |                              |
... abridged ...
| kube-system-kube-dns-9153     | Kubernetes | Accepted | svc name:      kube-dns      |
|                               |            |          | svc namespace: kube-system   |
|                               |            |          | port:          9153          |
|                               |            |          |                              |
+-------------------------------+------------+----------+------------------------------+

Notice in particular the default-httpbin-8000 Upstream that corresponds to the httpbin service we installed earlier.

Explore Function Discovery with OpenAPI

We could begin routing to this newly discovered httpbin Upstream right away. Before we do that, let’s explore advanced function discovery features that ship with open-source Gloo Edge. Function discovery is supported for both OpenAPI / REST and gRPC interfaces. In this example, we will associate an OpenAPI document with the httpbin Upstream and then observe the discovery feature at work.

We need to enable function discovery on the default namespace where the service is deployed. Since not all users employ OpenAPI or gRPC interfaces, and because function discovery can become resource-intensive, it is disabled by default. We could have enabled it via helm values at installation time; instead, we will do it here with kubectl:

kubectl label namespace default discovery.solo.io/function_discovery=enabled

Which confirms:

namespace/default labeled

Now we will modify the httpbin Upstream to associate an OpenAPI document. There’s nothing unique or Gloo-specific in the OpenAPI document itself; it’s just an OpenAPI spec for a standard REST interface. You can see the full spec for httpbin here and interact with the individual operations in the httpbin sandbox here.

Let’s take a look at the modifications to the generated httpbin Upstream. All we’re doing is adding a URL to the Upstream that locates the OpenAPI specification for the httpbin service.

    serviceSpec:
      rest:
        swaggerInfo:
          url: https://raw.githubusercontent.com/jameshbarton/solo-resources/main/zero-to-gateway/httpbin-openapi.json

Now we’ll apply this change to the Upstream:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-openapi-us.yaml

And it confirms:

upstream.gloo.solo.io/default-httpbin-8000 configured

Now when we use glooctl to inspect the upstream and compare it with what we had before, you can see that Gloo Edge has discovered (with the guidance of the OpenAPI document) a number of individual operations being published from the httpbin service. This will allow us to be much more precise and avoid errors as we establish routing rules.

glooctl get upstream default-httpbin-8000

You should see:

+----------------------+------------+----------+------------------------+
|       UPSTREAM       |    TYPE    |  STATUS  |        DETAILS         |
+----------------------+------------+----------+------------------------+
| default-httpbin-8000 | Kubernetes | Accepted | svc name:      httpbin |
|                      |            |          | svc namespace: default |
|                      |            |          | port:          8000    |
|                      |            |          | REST service:          |
|                      |            |          | functions:             |
|                      |            |          | - /anything            |
|                      |            |          | - /base64              |
|                      |            |          | - /brotli              |
|                      |            |          | - /bytes               |
|                      |            |          | - /cache               |
|                      |            |          | - /deflate             |
|                      |            |          | - /delay               |
|                      |            |          | - /delete              |
|                      |            |          | - /get                 |
|                      |            |          | - /gzip                |
|                      |            |          | - /headers             |
|                      |            |          | - /ip                  |
|                      |            |          | - /patch               |
|                      |            |          | - /post                |
|                      |            |          | - /put                 |
|                      |            |          | - /redirect-to         |
|                      |            |          | - /response-headers    |
|                      |            |          | - /status              |
|                      |            |          | - /stream              |
|                      |            |          | - /user-agent          |
|                      |            |          | - /uuid                |
|                      |            |          | - /xml                 |
|                      |            |          |                        |
+----------------------+------------+----------+------------------------+

CONTROL

In this section, we’ll establish and test routing rules that are the core of the proxy configuration, and also show how to establish timeout policies from the proxy.

Configure Simple Routing with CLI

Let’s begin our routing configuration with the simplest possible route to expose the /get operation on httpbin. This endpoint simply reflects back in its response the headers and any other arguments passed into the service.

We’ll use the glooctl utility to get started:

glooctl add route \
  --path-exact /api/httpbin/get \
  --dest-name default-httpbin-8000 \
  --prefix-rewrite /get

And can see the output:

{"level":"info","ts":"2021-06-08T13:34:28.582-0400","caller":"add/route.go:156","msg":"Created new default virtual service","virtualService":"virtual_host:{domains:\"*\"  routes:{matchers:{exact:\"/api/httpbin/get\"}  route_action:{single:{upstream:{name:\"default-httpbin-8000\"  namespace:\"gloo-system\"}}}  options:{prefix_rewrite:{value:\"/get\"}}}}  status:{}  metadata:{name:\"default\"  namespace:\"gloo-system\"  resource_version:\"437677\"  generation:1}"}
+-----------------+--------------+---------+------+---------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  | STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+---------+-----------------+----------------------------------+
| default         |              | *       | none | Pending |                 | /api/httpbin/get ->              |
|                 |              |         |      |         |                 | gloo-system.default-httpbin-8000 |
|                 |              |         |      |         |                 | (upstream)                       |
+-----------------+--------------+---------+------+---------+-----------------+----------------------------------+

This glooctl invocation created a Gloo Edge VirtualService component, which is named default by default. It routes any request to the path /api/httpbin/get to the httpbin /get endpoint. Attempting to reach any other endpoint on the httpbin service will be rejected.

Note that when the route is initially created, the status of the route is Pending. Issue the glooctl get virtualservice default command and observe that the status has now changed from Pending to Accepted.

+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| default         |              | *       | none | Accepted |                 | /api/httpbin/get ->              |
|                 |              |         |      |          |                 | gloo-system.default-httpbin-8000 |
|                 |              |         |      |          |                 | (upstream)                       |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+

Test the Simple Route with Curl

Now that a VirtualService is in place and a service endpoint is accessible from outside the application network, an external IP address is published from GKE.  We can see that on the gateway-proxy (Envoy proxy) service:

kubectl get service gateway-proxy -n gloo-system

Here is the response with the published EXTERNAL-IP:

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
gateway-proxy   LoadBalancer   10.7.242.211   34.75.42.228   80:32518/TCP,443:31657/TCP   8m32s

The glooctl utility provides an option to expose this as an alternative to fixed IP addresses, like this:

glooctl proxy url

And it responds with the http prefix of the exposed Gloo Edge URL:

http://34.75.42.228:80

We will now use this glooctl utility with curl to invoke httpbin endpoints.  In this case, we will add the curl -i option to show the HTTP response code and headers.

curl $(glooctl proxy url)/api/httpbin/get -i

This command should complete successfully.

HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 15:44:34 GMT
content-type: application/json
content-length: 281
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Original-Path": "/api/httpbin/get"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/get"
}

Note that if we attempt to invoke another valid endpoint /delay on the httpbin service, it will fail with a 404 Not Found error. Why? This is because our VirtualService routing policy is only exposing access to /get, one of the many endpoints available on the service. If we enter:

curl $(glooctl proxy url)/api/httpbin/delay/1 -i

You’ll see:

HTTP/1.1 404 Not Found
date: Fri, 18 Jun 2021 17:18:07 GMT
server: envoy
content-length: 0

Explore Complex Routing with Regex Patterns

Let’s assume that now we DO want to expose other httpbin endpoints like /delay. Our initial VirtualService is inadequate, because it is looking for an exact path match with /api/httpbin/get. Here is the core YAML for that VirtualService as constructed by the glooctl add route command we issued earlier.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - exact: /api/httpbin/get
      options:
        prefixRewrite: /get
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

This time, rather than using the glooctl CLI, let’s manipulate the VirtualService directly. We’ll modify the matchers: stanza to match the path prefix /api/httpbin and replace it with /. So a path like /api/httpbin/delay/1 will be sent to httpbin with the path /delay/1. Now it will look like this:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /api/httpbin
      options:
        regexRewrite:
          pattern:
            regex: '/api/httpbin/'
          substitution: '/'
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: gloo-system

Let’s apply the modified VirtualService and test. Note that throughout this exercise, we are managing Gloo Edge artifacts using Kubernetes utilities like kubectl. That’s an important point because it allows developers to work with familiar tools when working with Gloo Edge configuration. It also benefits organization using GitOps strategies to manage deployments, as tools like ArgoCD and Flux are able to easily handle Gloo artifacts as first-class Kubernetes citizens.

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-regex.yaml

Note that you can safely ignore the “kubectl apply” warning below. As long as kubectl responds that the default VirtualService was configured, then your change was applied.

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
virtualservice.gateway.solo.io/default configured

Test Routing with Regex Patterns

When we used only a single route with an exact match pattern, we could only exercise the httpbin /get endpoint. Let’s now use curl to confirm that both /get and /delay work as expected.

% curl $(glooctl proxy url)/api/httpbin/get -i
HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 17:27:01 GMT
content-type: application/json
content-length: 281
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Original-Path": "/api/httpbin/get"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/get"
}
% curl $(glooctl proxy url)/api/httpbin/delay/1 -i
HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 17:27:10 GMT
content-type: application/json
content-length: 335
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1004

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Original-Path": "/api/httpbin/delay/1"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/delay/1"
}

Perfect! It works just as expected! For extra credit, try out some of the other endpoints published via httpbin as well.

Configure Timeouts

Don’t you hate it when you visit a website and the request just gets “lost”? You wait and wait. Maybe you see the network connection established but then you wait some more, and still the request never completes.

Gloo Edge provides an easy-to-configure set of timeouts that you can apply to spare your valuable users this frustration. Like other Gloo features, it can be added to your policy with standard Kubernetes tooling, and without touching the source application. All we need to do add a timeout directive to our VirtualService. In this case, we will apply the timeout in the simplest fashion at the httpbin route level by adding timeout to our route options.

    routes:
    - matchers:
      - prefix: /api/httpbin
      options:
        timeout: '5s'  # Adding 5-second timeout HERE
        regexRewrite: 
          pattern:
            regex: '/api/httpbin/'
          substitution: '/'

Let’s apply this VirtualService change using kubectl:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-timeout.yaml

It confirms:

virtualservice.gateway.solo.io/default configured

Test Timeouts with httpbin delays

We will confirm that our new timeout policy works by using the httpbin /delay endpoint. First, we’ll specify a one second delay, and we expect everything to work just fine. Second, we’ll specify a longer delay, say eight seconds, and we will expect our timeout policy to be triggered and return a 504 Gateway Timeout error. Run:

curl $(glooctl proxy url)/api/httpbin/delay/1 -i

This returns:

HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 17:28:59 GMT
content-type: application/json
content-length: 334
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1003

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "5000",
    "X-Envoy-Original-Path": "/api/httpbin/delay/1"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/delay/1"
}

Note that the operation completed successfully and that the 1-second delay was applied. The response header x-envoy-upstream-service-time: 1006 indicates that the request spent 1,006 milliseconds being processed by Envoy.

Now let’s switch to an eight second delay and see if our five second timeout triggers as expected. Execute:

% curl $(glooctl proxy url)/api/httpbin/delay/8 -i
HTTP/1.1 504 Gateway Timeout
content-length: 24
content-type: text/plain
date: Fri, 18 Jun 2021 17:29:49 GMT
server: envoy

upstream request timeout

BOOM! Our simple timeout policy works just as expected, triggering a 504 Gateway Timeout error when the five second threshold is exceeded by our eight second httpbin delay.

DEBUG

Sometimes debugging bad software configurations can be a pain. Gloo Edge engineers have made this as easy as possible, with documentation like this. However, as we have all experienced, it can be a challenge with any complex system. Here we’ll explore how to use the glooctl utility to assist in some simple debugging tasks.

Solve a Problem with glooctl CLI

A common source of Gloo Edge configuration errors is mistyping an upstream reference, perhaps when copy/pasting it from another source but “missing a spot” when changing the name of the Upstream target. In this example, we’ll simulate making an error like that, and then demonstrating how glooctl can be used to detect it.

First, let’s apply a change to simulate the mistyping of an upstream config so that it is targeting a non-existent default-httpbin-8080 Upstream, rather than the correct default-httpbin-8000:

kubectl delete virtualservice default -n gloo-system
sleep 5
kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-bad-port.yaml

You should see:

virtualservice.gateway.solo.io "default" deleted
virtualservice.gateway.solo.io/default created

Note that we applied a sleep between deleting the VirtualService and re-creating it to ensure that we clear the working route from Envoy’s route cache before we create the new mis-configured route.

Now if we try to access one of our httpbin endpoints:

curl $(glooctl proxy url)/api/httpbin/get -i

It likely fails with an error like this:

curl: (7) Failed to connect to 34.75.42.228 port 80: Connection refused

So we’ll deploy one of the Gloo Edge debugging tools, the glooctl check utility. It performs checks on a number of Gloo resources, confirming that they are configured correctly and are interconnected with other resources correctly. For example, in this case, glooctl will detect the error in the mis-connection between a VirtualService and its Upstream target:

glooctl check

You can see the checks respond:

Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... 2 Errors!
Checking gateways... OK
Checking proxies... 1 Errors!
Error: 3 errors occurred:
	* Found virtual service with warnings: gloo-system default (Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.default-httpbin-8080 } not found)
	* Virtual service references unknown upstream: (Virtual service: gloo-system default | Upstream: gloo-system default-httpbin-8080)
	* Found proxy with warnings: gloo-system gateway-proxy
Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.default-httpbin-8080 } not found

The detected errors clearly identify that the VirtualService is pointed at an invalid destination.

So let’s reapply the previous configuration, and then we’ll confirm that the configuration is again clean:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/httpbin-vs-timeout.yaml

Now we get confirmation:

virtualservice.gateway.solo.io/default configured

Re-run glooctl check and observe that there are no problems. Our curl commands to the httpbin endpoint will also work again as expected:

Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... OK
Checking gateways... OK
Checking proxies... OK
No problems detected.

OBSERVE

Finally, let’s tackle an exercise where we’ll learn about some simple observability tools that ship with open-source Gloo Edge.

Configure Simple Access Logging

The default Envoy Proxy logging configurations are very quiet by design. When working in extremely high-volume environments, verbose logs can potentially impact performance and consume excessive storage.

However, access logs are quite important at development and test time, and potentially in production as well. So let’s first explore how to set up and consume simple access logs.

So far we have discussed Gloo Edge custom resources like Upstreams and VirtualServices. Upstreams represent the target systems to which Gloo routes traffic. VirtualServices represent the policies that determine how external requests are routed to those targets.

With access logging, we will consider another Gloo Edge component called a gateway. A gateway is a custom resource that configures the protocols and ports on which Gloo Edge listens for traffic. For example, by default Gloo Edge will have a gateway configured for HTTP and HTTPS traffic. More information on gateways is available here. Gloo Edge allows you to customize the behavior of your gateways in multiple ways. One of those ways is by adding access logs.

Let’s start by adding the simplest, default access log configuration. We’ll simply add the following options to activate access logging. You can see the full gateway YAML that we’ll apply here.

  options:
    accessLoggingService:
      accessLog:
      - fileSink:
          path: /dev/stdout
          stringFormat: ""

Now let’s apply this change to enable access logging from our Envoy data plane:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/gateway-basic-access-logs.yaml

As before, you can safely ignore the benign kubectl warning.

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
gateway.gateway.solo.io/gateway-proxy configured

Now let’s generate some traffic to produce some access logs.

curl $(glooctl proxy url)/api/httpbin/get
curl $(glooctl proxy url)/api/httpbin/delay/1
curl $(glooctl proxy url)/api/httpbin/delay/8

You should be able to view the resulting access logs by using the kubectl logs commands against the Envoy data plane pod:

kubectl logs -n gloo-system deploy/gateway-proxy

Here’s what we saw:

[2021-06-18T17:37:09.339Z] "GET /api/httpbin/get HTTP/1.1" 200 - 0 280 3 3 "-" "curl/7.64.1" "c3415777-d477-45ad-b51d-782663a4d818" "34.75.42.228" "10.4.2.9:80"
[2021-06-18T17:37:10.163Z] "GET /api/httpbin/delay/1 HTTP/1.1" 200 - 0 334 1002 1002 "-" "curl/7.64.1" "065099d6-61d1-4851-860d-20cefcc72ac6" "34.75.42.228" "10.4.2.9:80"
[2021-06-18T17:37:12.124Z] "GET /api/httpbin/delay/8 HTTP/1.1" 504 UT 0 24 5000 - "-" "curl/7.64.1" "c37c4551-ca93-4493-9e86-60222b936e3a" "34.75.42.228" "10.4.2.9:80"

Notice from the output the default string formatted access log for each of the operations we executed. You can see the paths of the operations, plus the HTTP response codes:  200 for the first two, and 504 (Gateway Timeout). Plus there is a host of other information.

While we are viewing these access logs using kubectl, you may want to export it for use with an enterprise log aggregator like ELK, Splunk, or Datadog. For example, Gloo Edge provides guidance for integrating with popular platforms like Datadog.

Customize Access Logging

Gloo Edge gateways can also be configured to produce access logs in other formats like JSON, and to customize the actual content that is published to those logs. We will do both of those things by replacing our previous access log configuration in the gateway component with this:

  options:
    accessLoggingService:
      accessLog:
      - fileSink:
          jsonFormat:
            # HTTP method name
            httpMethod: '%REQ(:METHOD)%'
            # Protocol. Currently either HTTP/1.1 or HTTP/2.
            protocol: '%PROTOCOL%'
            # HTTP response code. Note that a response code of ‘0’ means that the server never sent the
            # beginning of a response. This generally means that the (downstream) client disconnected.
            responseCode: '%RESPONSE_CODE%'
            # Total duration in milliseconds of the request from the start time to the last byte out
            clientDuration: '%DURATION%'
            # Total duration in milliseconds of the request from the start time to the first byte read from the upstream host
            targetDuration: '%RESPONSE_DURATION%'
            # Value of the "x-envoy-original-path" header (falls back to "path" header if not present)
            path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
            # Upstream cluster to which the upstream host belongs to
            upstreamName: '%UPSTREAM_CLUSTER%'
            # Request start time including milliseconds.
            systemTime: '%START_TIME%'
            # Unique tracking ID
            requestId: '%REQ(X-REQUEST-ID)%'
          path: /dev/stdout

More information on customizing access log content is provided here.

Now we will apply this gateway change, generate some additional traffic to our proxy, and view the resulting logs.

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/zero-to-gateway/gateway-json-access-logs.yaml

This responds:

gateway.gateway.solo.io/gateway-proxy configured

We’ll use the same curl commands as before to generate some traffic.

curl $(glooctl proxy url)/api/httpbin/get
curl $(glooctl proxy url)/api/httpbin/delay/1
curl $(glooctl proxy url)/api/httpbin/delay/8

Note that it may take a few seconds until the access log content is flushed to the logs:

kubectl logs -n gloo-system deploy/gateway-proxy | grep ^{ | jq

In the end, you should be able to see our customized JSON content, looking something like this:

{
  "protocol": "HTTP/1.1",
  "requestId": "56147cef-ca60-453b-a4c7-b7db039051a4",
  "responseCode": 200,
  "clientDuration": 3,
  "targetDuration": 3,
  "systemTime": "2021-06-18T17:38:50.530Z",
  "upstreamName": "default-httpbin-8000_gloo-system",
  "httpMethod": "GET",
  "path": "/api/httpbin/get"
}
{
  "path": "/api/httpbin/delay/1",
  "clientDuration": 1002,
  "targetDuration": 1002,
  "httpMethod": "GET",
  "protocol": "HTTP/1.1",
  "systemTime": "2021-06-18T17:38:51.313Z",
  "requestId": "138ea690-0798-43a7-acdb-d4db1c7f5f59",
  "upstreamName": "default-httpbin-8000_gloo-system",
  "responseCode": 200
}
{
  "protocol": "HTTP/1.1",
  "systemTime": "2021-06-18T17:38:53.088Z",
  "clientDuration": 5000,
  "targetDuration": null,
  "path": "/api/httpbin/delay/8",
  "httpMethod": "GET",
  "requestId": "bad18dff-1f33-42bd-bd17-e6f60bc92377",
  "upstreamName": "default-httpbin-8000_gloo-system",
  "responseCode": 504
}

Explore Envoy Metrics

Envoy publishes many metrics that may be useful for observing system behavior. In our very modest GKE cluster for this exercise, you can count over 3,000 individual metrics! You can learn more about them in the Envoy documentation here.

Let’s take a quick look at a couple of the useful metrics that Envoy produces for every one of our Upstream targets.

We’ll port-forward the Envoy administrative port 19000 to our local workstation:

kubectl port-forward -n gloo-system deploy/gateway-proxy 19000 &

This shows:

Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000

Then let’s view two of the metrics that are most relevant to this exercise: one that counts the number of successful (HTTP 200) requests processed by our httpbin Upstream, and another that counts the number of gateway timeout (HTTP 504) requests against that same upstream:

curl -s http://localhost:19000/stats | grep -E 'cluster.default-httpbin-8000_gloo-system.upstream_rq_(200|504)'

Which gives us:

cluster.default-httpbin-8000_gloo-system.upstream_rq_200: 7
cluster.default-httpbin-8000_gloo-system.upstream_rq_504: 2

As you can see, on this instance we’ve processed seven good requests and two bad ones. If we apply the same three curl requests as before, we’d expect the number of 200 requests to increment by two, and the number of 504 requests to increment by one:

curl $(glooctl proxy url)/api/httpbin/get
curl $(glooctl proxy url)/api/httpbin/delay/1
curl $(glooctl proxy url)/api/httpbin/delay/8
curl -s http://localhost:19000/stats | grep -E 'cluster.default-httpbin-8000_gloo-system.upstream_rq_(200|504)'

And that is exactly what we see!

cluster.default-httpbin-8000_gloo-system.upstream_rq_200: 9
cluster.default-httpbin-8000_gloo-system.upstream_rq_504: 3

If you’d like to have more tooling and enhanced visibility around system observability, we recommend taking a look at an Enterprise subscription to Gloo Edge. You can sign up for a free trial here.

Gloo Edge Enterprise provides out-of-the-box integration with both Prometheus and Grafana, allowing you to replace curl and grep with per-Upstream generated dashboards like this.

You can learn more about creating your own Grafana dashboards from Gloo Edge Enterprise metrics in this blog post.

CLEANUP

If you’d like to cleanup the work you’ve done, you can delete your GKE cluster from the Google Cloud console or by using the gcloud CLI:

gcloud container clusters delete my-edge-demo -q

You should see output similar to this if successful:

Deleting cluster my-edge-demo...
Deleting cluster my-edge-demo...done.
Deleted [https://container.googleapis.com/v1/projects/solo-test-236622/zones/us-east1-d/clusters/my-edge-demo].

Learn More

In this blog post, we explored how you can get started with the open-source edition of Gloo Edge in 15 minutes on your own workstation. We walked through the process of establishing access to a GKE cluster, installing an application, and then managing it with policies for routing, service discovery, timeouts, debugging, access logging, and observability. All of the code used in this guide is available on github.

A Gloo Edge Enterprise subscription offers even more value to users who require:

  • integration with identity management platforms like Auth0 and Okta;
  • configuration-driven rate limiting;
  • securing your application network with WAF, ModSecurity, or Open Policy Agent;
  • an API Portal for publishing and managing OpenAPI and gRPC interfaces; and
  • enhanced observability with batteries-included Prometheus and Grafana instances.
For more information, check out the following resources.