Running Envoy API Gateways on Kubernetes or Consul or both

The cloud-native community has converged on Envoy Proxy as a de-facto data plane proxy for application (L7) networking. You will find Envoy running as the data plane for many different service-mesh implementations as well as ingress and decentralized API Gateway solutions. One of the biggest reasons why Envoy is so well adopted these days is its open interface for a pluggable control plane.

Envoy uses its xDS interface to control just about every aspect of the proxy, dynamically, at run time. This means you can have a pluggable implementation for things like service discovery, routing, and security. To make this operational, you need to have a control plane that can plug into these various backends and serve up the correct Envoy configuration over xDS.

Gloo is an open-source API Gateway project from Solo.io that builds on Envoy Proxy and does exactly this: it provides a clean separation of concerns for the control plane and allows you to mold the control plane to your environment. No two environments and processes are exactly the same, so Gloo follows the “open-closed” principle where the core is closed for modifications, but we can enrich its capabilities by extending it with plugins. This approach is discussed at length in Guidance for Building a Control Plane to Manage Envoy.

One example of this extensibility is running Gloo in Kubernetes natively with CRDs (not hacky annotations) or outside of Kubernetes standalone with similar declarative configurations. We can leverage the Vault and Consul projects from our friends over at Hashicorp to make this happen. With Gloo as the control plane for Envoy, we can take advantage of these two important use cases:

1. Running an API Gateway within or outside of Kubernetes
2. Routing to services that live within or outside of Kubernetes

Use-case 2 is near and dear to Gloo — hence the name. Gloo is intending to provide L7 network control and observability to unify hybrid environments (running VMs, containers, and even FaaS/Lambda). Let’s look a bit closer at solving these two use cases with Consul and Vault.

Running our API Gateway outside of Kubernetes

Kubernetes affords us some great operational simplifications for Gloo (and applications in general). Kubernetes has a nice declarative configuration structure, service-discovery mechanism, storage and secret API, etc. If we opt to run outside of Kubernetes, we need to have a way to:

• define our declarative configuration
• store our declarative configuration
• store our secrets and keys

Using Gloo with Consul and Vault, we can accomplish this.

In this model, Gloo is configured to use Consul as a backing store for routing configuration and use Vault to store/access secrets (ie, keys needed to set up TLS, talk to Lambda, etc)

The user specifies their configuration in a declarative format (similar to Kubernetes resources) and Gloo reifies the configuration. When talking directly to Consul, a user could do something like:

curl -v 
    -XPUT 
    --data-binary "@<resource yaml file>.yaml" 
    "http://127.0.0.1:8500/v1/kv/gloo/<resource group>/<group version>/<resource kind>/<namespace>/<name>"

This will insert the Gloo configuration into Consul and Gloo will read it and present it to Envoy. Please see the documentation on Consul as a backend for more.

Routing to services living outside of Kubernetes

Another very important enterprise use case is the ability to use multiple different sources for service discovery. When running in Kubernetes, we get the baked-in service discovery, but organizations may be using Consul (or some other approach) to service discovery that we want to merry into our L7 application networking control (service mesh or API Gateway). With Gloo, we can configure Consul as a source for service discovery and route to those services.

The services registered with Consul can be running anywhere: VMs, containers, physical machines, etc. If running on EC2, btw, Gloo does support direct EC2 discovery as well.

For more on routing to Consul-registered services, please see the documentation.

Wrapping up

Envoy is a versatile L7 data plane insofar your control plane can support it. Gloo supports running natively within Kubernetes as well as outside of Kubernetes with the help of Consul and Vault. Please take a look at the Gloo project to see what a modern, decentralized API Gateway looks like built on Envoy with support for multiple platforms. Please reach out on Slack or Twitter and follow @christianposta @soloio_inc for more!