From Zero to Gloo Edge in 15 Minutes: ArgoCD GitOps Edition

My first real exposure to GitOps was while working on a project for an Amazon subsidiary where we used AWS technologies like CloudFormation. That was nice, but I yearned for something more general that wouldn’t lock me in to a single cloud provider.

My real GitOps “conversion” happened some time later when I was working as an architect on a 10-week digital modernization project for a US public-sector client. We had an ambitious goal and were feeling a bit of schedule pressure. One Monday morning we arrived at the client’s workspace only to discover that our entire Kubernetes development environment was mysteriously wiped out. The temptation was to try to understand what happened and fix it. But our project SREs had constructed everything using a still-new GitOps platform called Argo. Within minutes they had reconstructed our environment from the “spec” stored in our git repo. We avoided a very costly delay, and at least one new GitOps fanboy was born.

Fast-forward a few years, and now as a Field Engineer for solo.io, many clients ask us about best practices for using GitOps with the Gloo product line. It’s one of my favorite questions to hear, because it gives us an opportunity to discuss the strengths of our products’ cloud-native architecture. For example, the fact that Gloo configuration artifacts are expressed as YAML in Kubernetes Custom Resource Definitions (CRDs) means that they can be stored as artifacts in a git repo. That means they fit hand-in-glove with GitOps platforms like Flux and Argo Continuous Delivery (ArgoCD). It also means that they can be stored at runtime in native Kubernetes etcd storage. That means there are no external databases to configure with the Gloo Edge API Gateway.

Are you thinking of adopting an API gateway like Gloo Edge? Would you like to understand how that fits with popular GitOps platforms like Argo? Then this post is for you.

Give us 15 minutes, and we’ll give you a Kubernetes-hosted application accessible via an Envoy-based gateway configured with policies for routing, service discovery, timeouts, debugging, access logging, and observability. And we’ll manage the configuration entirely in a GitHub repo using the Argo GitOps platform.

Not interested in using Argo with Gloo Edge today? Not a problem. Check out one of the other installments in our “Getting Started in 15 Minutes” series:

cd argocd
./install-argocd.sh

No context specified. Using default context of "your-kube-context"
Beginning install on context "your-kube-context"....
namespace/argocd created
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
...
networkpolicy.networking.k8s.io/argocd-server-network-policy created
secret/argocd-secret patched

cd ..
./tools/wait-for-rollout.sh deployment argocd-server argocd 10

No context specified. Using current context of "your-kube-context"
Waiting 10 seconds for deployment argocd-server to come up.
Waiting for deployment "argocd-server" rollout to finish: 0 of 1 updated replicas are available...
deployment "argocd-server" successfully rolled out

kubectl port-forward svc/argocd-server -n argocd 8080:443

curl -sL https://run.solo.io/gloo/install | sh
export PATH=$HOME/.gloo/bin:$PATH

glooctl version

Client: {"version":"1.10.0"}
Server: version undefined, could not find any version of gloo running

glooctl upgrade --release=v1.10.0

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog-edge-argo-15/main/argo/gloo-edge-oss-helm-1-10-1.yaml

namespace/gloo-system created
application.argoproj.io/gloo-edge-oss-helm-1-10-1 created

until kubectl get ns gloo-system
do
sleep 1
done

until [ $(kubectl -n gloo-system get pods -o jsonpath='{range .items[*].status.containerStatuses[*]}{.ready}{"\n"}{end}' | grep false -c) -eq 0 ]; do
echo "Waiting for all the gloo-system pods to become ready"
sleep 1
done

echo "Gloo Edge deployment is ready :-)"

NAME          STATUS   AGE
gloo-system   Active   15s
Waiting for all the gloo-system pods to become ready
Waiting for all the gloo-system pods to become ready
Waiting for all the gloo-system pods to become ready
Gloo Edge deployment is ready :-)

gh repo fork https://github.com/solo-io/solo-blog-edge-argo-15.git --clone

- Forking solo-io/solo-blog-edge-argo-15...
✓ Created fork jameshbarton/solo-blog-edge-argo-15
Cloning into 'solo-blog-edge-argo-15'...
remote: Enumerating objects: 31, done.
remote: Counting objects: 100% (31/31), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 31 (delta 8), reused 26 (delta 6), pack-reused 0
Unpacking objects: 100% (31/31), done.
Updating upstream
From https://github.com/solo-io/solo-blog-edge-argo-15
 * [new branch]      main       -> upstream/main
✓ Cloned fork

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: httpbin-app
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    # Change repoURL to point to your clone of this repo
    # repoURL: https://github.com/your-github-account/solo-blog-edge-argo-15
    repoURL: https://github.com/solo-io/solo-blog-edge-argo-15
    targetRevision: HEAD
    path: cfg
  destination:
    server: https://kubernetes.default.svc
    namespace: httpbin
  # We're explicitly setting the syncPolicy to be empty, so we can use the
  # Argo UI to more easily see the impact of incremental changes in our configuration.  
  syncPolicy: {} 

kubectl apply -f argo/httpbin-argo-app.yaml

application.argoproj.io/httpbin-app created

kubectl get pods -l app=httpbin -n httpbin

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-66cdbdb6c5-2cnm7   1/1     Running   0          21m

DISCOVER ArgoCD and Gloo Edge

A unique feature of Gloo Edge is its ability to discover Kubernetes services and wrap them into an Upstream abstraction. Upstreams represent targets to which request traffic can be routed.  To learn more about how Upstreams operate in a Gloo Edge environment, see the product documentation here and here, and the API reference here.

Explore Service Discovery

Let’s use the glooctl utility to explore the catalog of Upstreams that Gloo Edge has already compiled within our Kubernetes cluster. You can run:

glooctl get upstreams

And you’ll see:

+-------------------------------------+------------+----------+--------------------------------+
|              UPSTREAM               |    TYPE    |  STATUS  |            DETAILS             |
+-------------------------------------+------------+----------+--------------------------------+
| argocd-argocd-dex-server-5556       | Kubernetes | Accepted | svc name:                      |
|                                     |            |          | argocd-dex-server              |
|                                     |            |          | svc namespace: argocd          |
|                                     |            |          | port:          5556            |
... snip ...
| httpbin-httpbin-8000                | Kubernetes | Accepted | svc name:      httpbin         |
|                                     |            |          | svc namespace: httpbin         |
|                                     |            |          | port:          8000            |

... snip ...
| kube-system-metrics-server-443      | Kubernetes | Accepted | svc name:      metrics-server  |
|                                     |            |          | svc namespace: kube-system     |
|                                     |            |          | port:          443             |
|                                     |            |          |                                |
+-------------------------------------+------------+----------+--------------------------------+

Notice in particular the httpbin-httpbin-8000 Upstream that corresponds to the httpbin service we installed earlier.

Explore Function Discovery with OpenAPI

We could begin routing to this newly discovered httpbin Upstream right away. Before we do that, let’s explore advanced function discovery features that ship with open-source Gloo Edge. Function discovery is supported for both OpenAPI / REST and gRPC interfaces. In this example, we will associate an OpenAPI document with the httpbin Upstream and then observe the discovery feature at work.

Enable function discovery on the namespace

We need to ensure that function discovery is enabled on the httpbin namespace where the service is deployed. Since not all users employ OpenAPI or gRPC interfaces, and because function discovery can become resource-intensive, it is disabled by default. We could have enabled it via helm values at installation time or with tools like kubectl operating directly on the cluster. But in the GitOps spirit of this post, we have simply stored a modified Namespace manifest to our GitHub repo and allowed the Argo controller to do the heavy lifting when it created the application. Note the label discovery.solo.io/function_discovery: enabled. That’s the key that unlocks Gloo Edge function discovery for us.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    discovery.solo.io/function_discovery: enabled
  name: httpbin
spec:
  finalizers:
  - kubernetes

You can see the modified Namespace YAML spec here.

Establish an OpenAPI spec to discover the published endpoints

Now we will modify the httpbin Upstream to associate an OpenAPI document. There’s nothing unique or Gloo-specific in the OpenAPI document itself; it’s just an OpenAPI spec for a standard REST interface. You can see the full spec for httpbin here and interact with the individual operations in the httpbin sandbox here.

Let’s take a look at the modifications to the generated httpbin Upstream. All we’re doing is adding a URL to the Upstream that locates the OpenAPI specification for the httpbin service.

    serviceSpec:
      rest:
        swaggerInfo:
          url: https://raw.githubusercontent.com/solo-io/solo-blog-edge-argo-15/main/lib/httpbin-openapi.json

We’ll copy the new Upstream manifest from our library directory to our cfg directory, which is what we configured our Argo Application to watch for configuration changes. Then we’ll commit those changes to our repo. Finally, we’ll use the argocd CLI to activate the sync process with the Argo controller. This is equivalent to the SYNC operation we performed in the UI earlier. Issue these commands from the root level of the forked blog repository:

cp lib/httpbin-openapi-us.yaml cfg 
git add . 
git commit -m "add openapi spec to upstream for function discovery" -a 
git push origin main 
argocd app sync httpbin-app

Note that if you are not logged into Argo from the command-line, the argocd app sync may not work for you. Use argocd login first to authenticate yourself:

argocd login localhost:8080

Provide the credentials as with the UI earlier (username admin, password solo.io), and argocd should establish a session for you.

WARNING: server certificate had error: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? y
Username: admin
Password:
'admin:login' logged in successfully
Context 'localhost:8080' updated

If the argocd sync command failed for you earlier, you should be able to re-run argocd app sync httpbin-app now to ensure that the OpenAPI changes have been deployed.

Now when we use glooctl to inspect the upstream and compare it with what we had before, you can see that Gloo Edge has discovered (with the guidance of the OpenAPI document) a number of individual operations being published from the httpbin service. This will allow us to be much more precise and avoid errors as we establish routing rules.

glooctl get upstream httpbin-httpbin-8000

You should see:

+----------------------+------------+----------+------------------------+
|       UPSTREAM       |    TYPE    |  STATUS  |        DETAILS         |
+----------------------+------------+----------+------------------------+
| httpbin-httpbin-8000 | Kubernetes | Accepted | svc name:      httpbin |
|                      |            |          | svc namespace: httpbin |
|                      |            |          | port:          8000    |
|                      |            |          | REST service:          |
|                      |            |          | functions:             |
|                      |            |          | - /anything            |
|                      |            |          | - /base64              |
|                      |            |          | - /brotli              |
|                      |            |          | - /bytes               |
|                      |            |          | - /cache               |
|                      |            |          | - /deflate             |
|                      |            |          | - /delay               |
|                      |            |          | - /delete              |
|                      |            |          | - /get                 |
|                      |            |          | - /gzip                |
|                      |            |          | - /headers             |
|                      |            |          | - /ip                  |
|                      |            |          | - /patch               |
|                      |            |          | - /post                |
|                      |            |          | - /put                 |
|                      |            |          | - /redirect-to         |
|                      |            |          | - /response-headers    |
|                      |            |          | - /status              |
|                      |            |          | - /stream              |
|                      |            |          | - /user-agent          |
|                      |            |          | - /uuid                |
|                      |            |          | - /xml                 |
|                      |            |          |                        |
+----------------------+------------+----------+------------------------+

CONTROL your routing rules with ArgoCD GitOps

In this section, we’ll establish and test routing rules that are the core of the proxy configuration, and also show how to establish timeout policies from the proxy.

Configure Simple Routing with CLI

Let’s begin our routing configuration with the simplest possible route to expose the /get operation on httpbin. This endpoint simply reflects back in its response the headers and any other arguments passed into the service.

We will introduce a VirtualService into our configuration to define this route. VirtualServices are a central abstraction in Gloo Edge that manages routing policies. You can learn more about them and how they interact with other Gloo Edge components here.

Below is our starting-point VirtualService. It matches requests for any domain with a request path of /api/httpbin/get, rewrites the request to /get, and then forwards it to the httpbin Upstream service.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - exact: /api/httpbin/get
      options:
        prefixRewrite: /get
      routeAction:
        single:
          upstream:
            name: httpbin-httpbin-8000
            namespace: gloo-system

Issue these commands to move the VirtualService spec to the live configuration directory, and then commit that change to the repo.

cp lib/httpbin-vs-original.yaml cfg 
git add . 
git commit -m "initial virtualservice" -a 
git push origin main

If you REFRESH the Argo httpbin-app display, you can see that the new VirtualService httpbin-vs has been detected in the repo. (Isn’t it nice that all Gloo Edge abstractions like VirtualServices are expressed as Kubernetes Custom Resources, so that they are all first-class citizens in our Argo UI?)  But our VirtualService is flagged as being OutOfSync. When you initiate a SYNC from this UI (or the command-line), the app and the VirtualService will turn green as it is applied to our Kubernetes cluster.

We can also confirm using glooctl that our route has been accepted and applied to the proxy. Issue the glooctl get virtualservice httpbin-vs command and observe that the status is now Accepted.

+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES              |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+
| httpbin-vs      |              | *       | none | Accepted |                 | /api/httpbin/get ->              |
|                 |              |         |      |          |                 | gloo-system.httpbin-httpbin-8000 |
|                 |              |         |      |          |                 | (upstream)                       |
+-----------------+--------------+---------+------+----------+-----------------+----------------------------------+

Test the Simple Route with Curl

Now that a VirtualService is in place and a service endpoint is accessible from outside the application network, an external IP address is published from GKE.  We can see that on the gateway-proxy (Envoy proxy) service:

kubectl get service gateway-proxy -n gloo-system

Here is the response with the published EXTERNAL-IP:

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
gateway-proxy   LoadBalancer   10.7.242.211   34.75.42.228   80:32518/TCP,443:31657/TCP   8m32s

The glooctl utility provides an option to expose this as an alternative to fixed IP addresses, like this:

glooctl proxy url

And it responds with the http prefix of the exposed Gloo Edge URL:

http://34.75.42.228:80

We will now use this glooctl utility with curl to invoke httpbin endpoints.  In this case, we will add the curl -i option to show the HTTP response code and headers.

curl $(glooctl proxy url)/api/httpbin/get -i

This command should complete successfully.

HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 15:44:34 GMT
content-type: application/json
content-length: 281
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Original-Path": "/api/httpbin/get"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/get"
}

Note that if we attempt to invoke another valid endpoint /delay on the httpbin service, it will fail with a 404 Not Found error. Why? This is because our VirtualService routing policy is only exposing access to /get, one of the many endpoints available on the service. If we enter:

curl $(glooctl proxy url)/api/httpbin/delay/1 -i

You’ll see:

HTTP/1.1 404 Not Found
date: Fri, 18 Jun 2021 17:18:07 GMT
server: envoy
content-length: 0

Explore Complex Routing with Regex Patterns

Let’s assume that now we DO want to expose other httpbin endpoints like /delay. Our initial VirtualService is inadequate, because it is looking for an exact path match with /api/httpbin/get. As a reminder, here is the YAML that we used for the original VirtualService.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - exact: /api/httpbin/get
      options:
        prefixRewrite: /get
      routeAction:
        single:
          upstream:
            name: httpbin-httpbin-8000
            namespace: gloo-system

We’ll modify the matchers: stanza to not use an exact match on /api/httpbin/get. Instead, we will match the path prefix /api/httpbin and replace it with / before forwarding the request to the service. So a path like /api/httpbin/delay/1 will be sent to httpbin with the path /delay/1. Now it will look like this:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /api/httpbin
      options:
        regexRewrite:
          pattern:
            regex: '/api/httpbin/'
          substitution: '/'
      routeAction:
        single:
          upstream:
            name: httpbin-httpbin-8000
            namespace: gloo-system

Let’s use Argo to apply the modified VirtualService and test. The script below removes the original VirtualService YAML from our live configuration directory and replaces it with the one above. It then commits and pushes those changes to GitHub. Finally, it uses the argocd CLI to force a sync on the httpbin-app.

rm cfg/httpbin-vs-original.yaml
cp lib/httpbin-vs-regex.yaml cfg 
git add . 
git commit -m "add openapi spec to upstream for function discovery" -a 
git push origin main 
argocd app sync httpbin-app

Test Routing with Regex Patterns

When we used only a single route with an exact match pattern, we could only exercise the httpbin /get endpoint. Let’s now use curl to confirm that both /get and /delay work as expected.

% curl $(glooctl proxy url)/api/httpbin/get -i
HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 17:27:01 GMT
content-type: application/json
content-length: 281
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Original-Path": "/api/httpbin/get"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/get"
}
% curl $(glooctl proxy url)/api/httpbin/delay/1 -i
HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 17:27:10 GMT
content-type: application/json
content-length: 335
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1004

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Original-Path": "/api/httpbin/delay/1"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/delay/1"
}

Perfect! It works just as expected! For extra credit, try out some of the other endpoints published via httpbin as well.

Configure Timeouts

Don’t you hate it when you visit a website and the request just gets “lost”? You wait and wait. Maybe you see the network connection established but then you wait some more, and still the request never completes.

Gloo Edge provides an easy-to-configure set of timeouts that you can apply to spare your valuable users this frustration. Like other Gloo features, it can be added to your policy with standard Kubernetes tooling, and without touching the source application. All we need to do add a timeout directive to our VirtualService. In this case, we will apply the timeout in the simplest fashion at the httpbin route level by adding timeout to our route options.

    routes:
    - matchers:
      - prefix: /api/httpbin
      options:
        timeout: '5s'  # Adding 5-second timeout HERE
        regexRewrite: 
          pattern:
            regex: '/api/httpbin/'
          substitution: '/'

Let’s apply this VirtualService change using the git and argocd utilities:

rm cfg/httpbin-vs-regex.yaml
cp lib/httpbin-vs-timeout.yaml cfg 
git add . 
git commit -m "add timeout to httpbin VS" -a 
git push origin main 
argocd app sync httpbin-app

Test Timeouts with httpbin delays

We will confirm that our new timeout policy works by using the httpbin /delay endpoint. First, we’ll specify a one second delay, and we expect everything to work just fine. Second, we’ll specify a longer delay, say eight seconds, and we will expect our timeout policy to be triggered and return a 504 Gateway Timeout error. Run:

curl $(glooctl proxy url)/api/httpbin/delay/1 -i

This returns:

HTTP/1.1 200 OK
server: envoy
date: Fri, 18 Jun 2021 17:28:59 GMT
content-type: application/json
content-length: 334
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1003

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "34.75.42.228",
    "User-Agent": "curl/7.64.1",
    "X-Envoy-Expected-Rq-Timeout-Ms": "5000",
    "X-Envoy-Original-Path": "/api/httpbin/delay/1"
  },
  "origin": "10.4.1.4",
  "url": "http://34.75.42.228/delay/1"
}

Note that the operation completed successfully and that the 1-second delay was applied. The response header x-envoy-upstream-service-time: 1003 indicates that the request spent 1,003 milliseconds being processed by Envoy.

Now let’s switch to an eight second delay and see if our five second timeout triggers as expected. Execute:

% curl $(glooctl proxy url)/api/httpbin/delay/8 -i
HTTP/1.1 504 Gateway Timeout
content-length: 24
content-type: text/plain
date: Fri, 18 Jun 2021 17:29:49 GMT
server: envoy

upstream request timeout

BOOM! Our simple timeout policy works just as expected, triggering a 504 Gateway Timeout error when the five-second threshold is exceeded by our eight-second httpbin delay.

DEBUG Gloo Edge configuration

Sometimes debugging bad software configurations can be a pain. Gloo Edge engineers have made this as easy as possible, with documentation like this. However, as we have all experienced, it can be a challenge with any complex system. Here we’ll explore how to use the glooctl utility to assist in some simple debugging tasks.

Solve a Problem with glooctl CLI

A common source of Gloo Edge configuration errors is mistyping an upstream reference, perhaps when copy/pasting it from another source but “missing a spot” when changing the name of the Upstream target. In this example, we’ll simulate making an error like that, and then demonstrating how glooctl can be used to detect it.

First, let’s apply a change to simulate the mistyping of an upstream config so that it is targeting a non-existent httpbin-httpbin-8080 Upstream, rather than the correct httpbin-httpbin-8000. Note that we are doing this with kubectl to make changes directly to the cluster. Since we have Argo configured for manual sync, it will not immediately override our changes.

kubectl delete virtualservice httpbin-vs -n gloo-system
sleep 5
kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog-edge-argo-15/main/lib/httpbin-vs-bad-port.yaml

You should see:

virtualservice.gateway.solo.io "httpbin-vs" deleted
virtualservice.gateway.solo.io/httpbin-vs created

Note that we applied a sleep between deleting the VirtualService and re-creating it to ensure that we clear the working route from Envoy’s route cache before we create the new mis-configured route.

Now if we try to access one of our httpbin endpoints:

curl $(glooctl proxy url)/api/httpbin/get -i

It likely fails with an error like this:

curl: (7) Failed to connect to 34.75.42.228 port 80: Connection refused

So we’ll deploy one of the Gloo Edge debugging tools, the glooctl check utility. It performs checks on a number of Gloo resources, confirming that they are configured correctly and are interconnected with other resources correctly. For example, in this case, glooctl will detect the error in the mis-connection between a VirtualService and its Upstream target:

glooctl check

You can see the checks respond:

Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... 2 Errors!
Checking gateways... OK
Checking proxies... 1 Errors!
Error: 3 errors occurred:
	* Found virtual service with warnings: gloo-system default (Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.httpbin-httpbin-8080 } not found)
	* Virtual service references unknown upstream: (Virtual service: gloo-system default | Upstream: gloo-system httpbin-httpbin-8080)
	* Found proxy with warnings: gloo-system gateway-proxy
Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.httpbin-httpbin-8080 } not found

The detected errors clearly identify that the VirtualService is pointed at an invalid destination.

So let’s use argocd to restore the previous configuration using the saved state in our GitHub repo, and then we’ll confirm that the configuration is again clean:

argocd app sync httpbin-app

When that process completes, re-run glooctl check and observe that there are no problems. Our curl commands to the httpbin endpoint will also work again as expected:

Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking secrets... OK
Checking virtual services... OK
Checking gateways... OK
Checking proxies... OK
No problems detected.

OBSERVE your API gateway in action

Finally, let’s tackle an exercise where we’ll learn about some simple observability tools that ship with open-source Gloo Edge.

Configure Simple Access Logging

The default Envoy Proxy logging configurations are very quiet by design. When working in extremely high-volume environments, verbose logs can potentially impact performance and consume excessive storage.

However, access logs are quite important at development and test time, and potentially in production as well. So let’s first explore how to set up and consume simple access logs.

So far we have discussed Gloo Edge custom resources like Upstreams and VirtualServices. Upstreams represent the target systems to which Gloo routes traffic. VirtualServices represent the policies that determine how external requests are routed to those targets.

With access logging, we will consider another Gloo Edge component called a Gateway. A Gateway is a custom resource that configures the protocols and ports on which Gloo Edge listens for traffic. For example, by default Gloo Edge will have a gateway configured for HTTP and HTTPS traffic. More information on gateways is available here. Gloo Edge allows you to customize the behavior of your gateways in multiple ways. One of those ways is by adding access logs.

Let’s start by adding the simplest, default access log configuration. We’ll simply add the following options to activate access logging. You can see the full gateway YAML that we’ll apply here.

  options:
    accessLoggingService:
      accessLog:
      - fileSink:
          path: /dev/stdout
          stringFormat: ""

Disable Auto-Sync on Edge Argo App

We need to update a Gateway Custom Resource in order to apply the access log changes in this section. The issue is that we have installed our Gloo Edge Argo Application to auto-sync the underlying resources with the GitHub repo. While that’s likely a great idea in a production environment, it interferes with our ability to experiment with changes in this blog exercise.

To work around that, visit the gloo-edge-oss-helm-1-10-1 App Details page in the Argo UI. Scroll down and click the DISABLE AUTO-SYNC button and confirm on the resulting dialog. That will allow us to experiment without the Argo controller instantly overwriting our changes.

Apply and Test Gateway Changes Via Argo

Let’s apply our change to the default http gateway in order to enable access logging from our Envoy data plane. This time, we’ll use git to update the repo and argocd to trigger a sync of the repo with the cluster:

cp lib/gateway-basic-access-logs.yaml cfg 
git add . 
git commit -m "add basic access logging" -a 
git push origin main 
argocd app sync httpbin-app

Now let’s generate some traffic to produce some access logs.

curl $(glooctl proxy url)/api/httpbin/get
curl $(glooctl proxy url)/api/httpbin/delay/1
curl $(glooctl proxy url)/api/httpbin/delay/8

You should be able to view the resulting access logs by using the kubectl logs commands against the Envoy data plane pod:

kubectl logs -n gloo-system deploy/gateway-proxy

Here’s what we saw:

[2021-06-18T17:37:09.339Z] "GET /api/httpbin/get HTTP/1.1" 200 - 0 280 3 3 "-" "curl/7.64.1" "c3415777-d477-45ad-b51d-782663a4d818" "34.75.42.228" "10.4.2.9:80"
[2021-06-18T17:37:10.163Z] "GET /api/httpbin/delay/1 HTTP/1.1" 200 - 0 334 1002 1002 "-" "curl/7.64.1" "065099d6-61d1-4851-860d-20cefcc72ac6" "34.75.42.228" "10.4.2.9:80"
[2021-06-18T17:37:12.124Z] "GET /api/httpbin/delay/8 HTTP/1.1" 504 UT 0 24 5000 - "-" "curl/7.64.1" "c37c4551-ca93-4493-9e86-60222b936e3a" "34.75.42.228" "10.4.2.9:80"

Notice from the output the default string formatted access log for each of the operations we executed. You can see the paths of the operations, plus the HTTP response codes: 200 for the first two, and 504 (Gateway Timeout). Plus there is a host of other information.

While we are viewing these access logs using kubectl, you may want to export it for use with an enterprise log aggregator like ELK, Splunk, or Datadog. For example, Gloo Edge provides guidance for integrating with popular platforms like Datadog.

Customize Access Logging

Gloo Edge gateways can also be configured to produce access logs in other formats like JSON, and to customize the actual content that is published to those logs. We will do both of those things by replacing our previous access log configuration in the gateway component with this:

  options:
    accessLoggingService:
      accessLog:
      - fileSink:
          jsonFormat:
            # HTTP method name
            httpMethod: '%REQ(:METHOD)%'
            # Protocol. Currently either HTTP/1.1 or HTTP/2.
            protocol: '%PROTOCOL%'
            # HTTP response code. Note that a response code of ‘0’ means that the server never sent the
            # beginning of a response. This generally means that the (downstream) client disconnected.
            responseCode: '%RESPONSE_CODE%'
            # Total duration in milliseconds of the request from the start time to the last byte out
            clientDuration: '%DURATION%'
            # Total duration in milliseconds of the request from the start time to the first byte read from the upstream host
            targetDuration: '%RESPONSE_DURATION%'
            # Value of the "x-envoy-original-path" header (falls back to "path" header if not present)
            path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
            # Upstream cluster to which the upstream host belongs to
            upstreamName: '%UPSTREAM_CLUSTER%'
            # Request start time including milliseconds.
            systemTime: '%START_TIME%'
            # Unique tracking ID
            requestId: '%REQ(X-REQUEST-ID)%'
          path: /dev/stdout

More information on customizing access log content is provided here.

Now let’s use Argo to apply this gateway change, generate some additional traffic to our proxy, and view the resulting logs.

rm cfg/gateway-basic-access-logs.yaml
cp lib/gateway-json-access-logs.yaml cfg 
git add . 
git commit -m "use json access logs" -a 
git push origin main 
argocd app sync httpbin-app

When that process completes, we’ll use the same curl commands as before to generate some traffic.

curl $(glooctl proxy url)/api/httpbin/get
curl $(glooctl proxy url)/api/httpbin/delay/1
curl $(glooctl proxy url)/api/httpbin/delay/8

Note that it may take a few seconds until the access log content is flushed to the logs:

kubectl logs -n gloo-system deploy/gateway-proxy | grep ^{ | jq

In the end, you should be able to see our customized JSON content, looking something like this:

{
  "protocol": "HTTP/1.1",
  "requestId": "56147cef-ca60-453b-a4c7-b7db039051a4",
  "responseCode": 200,
  "clientDuration": 3,
  "targetDuration": 3,
  "systemTime": "2021-06-18T17:38:50.530Z",
  "upstreamName": "httpbin-httpbin-8000_gloo-system",
  "httpMethod": "GET",
  "path": "/api/httpbin/get"
}
{
  "path": "/api/httpbin/delay/1",
  "clientDuration": 1002,
  "targetDuration": 1002,
  "httpMethod": "GET",
  "protocol": "HTTP/1.1",
  "systemTime": "2021-06-18T17:38:51.313Z",
  "requestId": "138ea690-0798-43a7-acdb-d4db1c7f5f59",
  "upstreamName": "httpbin-httpbin-8000_gloo-system",
  "responseCode": 200
}
{
  "protocol": "HTTP/1.1",
  "systemTime": "2021-06-18T17:38:53.088Z",
  "clientDuration": 5000,
  "targetDuration": null,
  "path": "/api/httpbin/delay/8",
  "httpMethod": "GET",
  "requestId": "bad18dff-1f33-42bd-bd17-e6f60bc92377",
  "upstreamName": "httpbin-httpbin-8000_gloo-system",
  "responseCode": 504
}

Explore Envoy Metrics

Envoy publishes many metrics that may be useful for observing system behavior. In our very modest kind cluster for this exercise, you can count over 3,000 individual metrics! You can learn more about them in the Envoy documentation here.

Let’s take a quick look at a couple of the useful metrics that Envoy produces for every one of our Upstream targets.

We’ll port-forward the Envoy administrative port 19000 to our local workstation:

kubectl port-forward -n gloo-system deploy/gateway-proxy 19000 &

This shows:

Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000

Then let’s view two of the metrics that are most relevant to this exercise: one that counts the number of successful (HTTP 200) requests processed by our httpbin Upstream, and another that counts the number of gateway timeout (HTTP 504) requests against that same upstream:

curl -s http://localhost:19000/stats | grep -E 'cluster.httpbin-httpbin-8000_gloo-system.upstream_rq_(200|504)'

Which gives us:

cluster.httpbin-httpbin-8000_gloo-system.upstream_rq_200: 7
cluster.httpbin-httpbin-8000_gloo-system.upstream_rq_504: 2

As you can see, on this instance we’ve processed seven good requests and two bad ones. If we apply the same three curl requests as before, we’d expect the number of 200 requests to increment by two, and the number of 504 requests to increment by one:

curl $(glooctl proxy url)/api/httpbin/get
curl $(glooctl proxy url)/api/httpbin/delay/1
curl $(glooctl proxy url)/api/httpbin/delay/8

curl -s http://localhost:19000/stats | grep -E 'cluster.httpbin-httpbin-8000_gloo-system.upstream_rq_(200|504)'

And that is exactly what we see!

cluster.httpbin-httpbin-8000_gloo-system.upstream_rq_200: 9
cluster.httpbin-httpbin-8000_gloo-system.upstream_rq_504: 3

If you’d like to have more tooling and enhanced visibility around system observability, we recommend taking a look at an Enterprise subscription to Gloo Edge. You can sign up for a free trial here.

Gloo Edge Enterprise provides out-of-the-box integration with both Prometheus and Grafana, allowing you to replace curl and grep with per-Upstream generated dashboards like this.

You can learn more about creating your own Grafana dashboards from Gloo Edge Enterprise metrics in this blog post.

CLEANUP

If you’d like to cleanup the work you’ve done, you can simply delete your Argo Applications using its web UI or the argocd CLI.

Learn More