Deploying Envoy proxies outside Kubernetes with Gloo Edge

Some things should always be separated, shouldn’t they? Like oil and water. Wheat and chaff. Church and state. And how about deploying Envoy proxies and their control planes, like Gloo Edge?

By default, you might think of them as living together. In the Gloo Edge documentation, the default scenario deploys an Envoy proxy and its control plane to one Kubernetes cluster, which is how many users deploy both in development and production.

Deploying Envoy proxies outside Kubernetes

But there’s a possibility that you will need to deploy the Envoy proxies and their control planes differently. Perhaps you’d like to safely ensconce your control plane behind a firewall in its Kubernetes cluster. Maybe you’d like to deploy the Envoy proxies of your data plane separately, such as in a DMZ (demilitarized zone) on virtual machines. Separating deployments is a common request from enterprise users, especially those with large fleets of legacy assets that existed before Kubernetes was on anyone’s radar.

Can Gloo Edge do that? Absolutely!

In this blog post, we’ll walk through a separate-deployment scenario in which Gloo Edge allows you to deploy Envoy proxies outside of Kubernetes. The outlined steps show you how to first deploy the control plane to an Amazon Elastic Kubernetes Service (EKS) cluster, and then deploy the data plane to a non-Kubernetes Amazon EC2 virtual machine.

An architectural foundation for deploying Envoy proxies with Gloo Edge

Gloo Edge is built with a cloud-native architecture. This means that it’s deployed as a fleet of microservices, commonly within a single Kubernetes cluster. Typically, the Envoy proxies that serve as the data plane exist within the same cluster as the Edge control plane components that collaborate to dynamically serve configuration to Envoy. To learn more about the various control plane components, see the Gloo Edge documentation.

The key interface when deploying a managed Envoy proxy outside a Kubernetes cluster is the Envoy-to-Gloo connection. In almost all enterprise deployments, Envoy depends on a management server that generates configuration dynamically as policies and components in the cluster change. That management server adheres to an Envoy protocol called xDS (all Discovery Services). The gloo service acts as the xDS server to one or more Envoy proxies.

A default Gloo Edge installation implicitly assumes that the Envoy proxy, the gateway-proxy service, is deployed in the same cluster as its xDS server, the gloo service. Consequently, we’ll ensure for our purposes that the gloo service is established as a LoadBalancer service instead of its default ClusterIP type. That will allow the Envoy gateway-proxy to talk to its gloo xDS server in order to receive its configuration.

Deploying Envoy proxies step by step

To get started, we’ll establish a Kubernetes-deployed Gloo Edge control plane that manages a non-Kubernetes Envoy proxy. These steps use an Amazon EKS cluster to host the control plane and an out-of-cluster EC2 instance to host the standalone Envoy proxy. We’ll also pay special attention to the Gloo Edge bits of the configuration process.

Here’s a summary of the steps in the process:

  1. Provision AWS components, specifically an EKS cluster and an EC2 instance.
  2. Configure Gloo Edge Enterprise on the EKS cluster.
  3. Ensure gloo xDS server is available outside the cluster.
  4. Configure an external target service for testing.
  5. Configure a Gloo Edge VirtualService.
  6. Test with the internal Envoy proxy.
  7. Configure external Envoy on the EC2 instance.
  8. Test with the external Envoy proxy.

1. Provision AWS components

The general principles outlined in this post should work with any cloud provider. But in this case, we’ll work with AWS, using both an EKS cluster and a standalone EC2 instance. This basic approach is also known to work with Google GKE clusters.

Establish an EKS cluster

First, we’ll assume that you have access to an EKS cluster, or sufficient AWS permissions to create a cluster. If you don’t have the proper access, check out this blog post to get started with Gloo Edge on EKS.

If you’re creating a cluster, you can use the AWS console or the eksctl CLI. We used the eksctl CLI utility as a simpler alternative to the console. Try this command with the eksctl CLI to create a modest demo cluster with two m5.large instances for this scenario:

eksctl create cluster --name non-kube-envoy --region us-east-2

This command creates a cluster in the Ohio (us-east-2) region. If you don’t specify a region, eksctl defaults to the Oregon (us-west-2) region. A full AWS region list with corresponding codes is available here.

Configure an EC2 instance

Second, we created a standalone EC2 instance outside the EKS cluster to host our non-Kubernetes Envoy proxy. This was an m5.large instance  in the us-east-2 region. If you don’t have experience provisioning a standalone EC2, you might find the articles here and here helpful.

After you establish SSH (Secure Shell) access to your EC2 instance, you will need to install docker tools, including docker-compose. We used the following sequence on our EC2 instance:

$ sudo yum install docker
$ sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
# Confirm installation
$ docker-compose version
Docker Compose version v2.0.1

At this point, you should have an EKS cluster that allows access from remote addresses. You should also have an EC2 instance that contains basic docker tools and allows external SSH and HTTP access to facilitate testing.

2. Configure Gloo Edge Enterprise

We will use the Enterprise edition of Gloo Edge in these steps. The open source edition is also sufficient, with only minor configuration tweaks. However, note that in the forthcoming blog post for this scenario, which shows you how to extend the data plane deployment to include rate limiting and external auth servers, a Gloo Edge Enterprise license is required. You can request an Enterprise license key here.

If you haven’t already installed a Gloo Edge instance on a cluster, follow this guide to install the glooctl CLI and the Gloo Edge Enterprise server in a Kubernetes cluster. Note that we used Gloo Edge Enterprise version 1.9.2 for this scenario.

3. Ensure gloo xDS is available outside your cluster

As mentioned earlier, Gloo Edge typically assumes that the Envoy proxy service is deployed in the same cluster as its xDS server. That is not the case in this scenario, in which we are deploying Envoy outside of the Kubernetes cluster. Consequently, we need to change the default service type of the gloo service, since Envoy (gateway-proxy) needs to connect with it directly to receive its configuration.  To achieve that, we can patch the gloo service to change its service type from the default ClusterIP to LoadBalancer.

Once that change takes effect, AWS creates a CLB (Classic Load Balancer) instance that we can direct our Envoy proxy to use. Though we use the default CLB in this exercise, we also tested this successfully with an AWS NLB (Network Load Balancer). NLB is the more modern technology, and multiple Gloo Edge users have successfully configured it in production deployments.

kubectl patch svc gloo -n gloo-system -p '{"spec": {"type": "LoadBalancer"}}'

You should see a response like this:

service/gloo patched

To confirm that this patch worked as expected, first inspect the service type:

kubectl get svc gloo -n gloo-system -o=jsonpath='{.spec.type}'

In the ouput, verify that gloo is now a LoadBalancer service.

Next, verify that AWS has provisioned a proper elb hostname as the gloo external hostname:

kubectl get svc gloo -n gloo-system -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}'

You should see an AWS load balancer hostname, for example:

a7788fc21d90a4c3eb7e231344772a69-1799697922.us-east-2.elb.amazonaws.com

4. Configure the external target service

In this deployment pattern, the target service must either exist outside of a Kubernetes cluster or be routed to through an external endpoint. Keep in mind that the Envoy proxy that will handle this request is on an external VM, so it has no direct access to Kubernetes Services. We will simulate that scenario by creating an Upstream for the public httpbin service. This offers the added benefit of allowing us to test a number of different HTTP request types using a single service.

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: httpbin-us
  namespace: gloo-system
spec:
  static:
    hosts:
      - addr: httpbin.org
        port: 80

Add this Upstream to your Gloo Edge configuration:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/non-kube-envoy/httpbin-us.yaml

5. Configure the Edge virtual service

We will configure an Edge VirtualService to specify the routing rules that will direct requests to the external httpbin service.

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: httpbin-us
            namespace: gloo-system

Add this VirtualService to your Gloo Edge configuration:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/non-kube-envoy/httpbin-vs-ext.yaml

6. Test with internal Envoy proxy

Now that we defined both a VirtualService and an Upstream for an external service, we can confirm that everything is working by invoking the standard Envoy proxy deployed by Gloo Edge. (Note that if we were planning to take a non-Kubernetes Envoy deployment to production, we would disable this standard internal proxy.)

Once we established the valid Upstream and VirtualService, EKS automatically created an AWS Classic Load Balancer (CLB) in front of the internal Envoy proxy. You can find the hostname and URL associated with that proxy by using this glooctl command:

% glooctl proxy url
http://ac2440158914049ba8880519a5234c6c-2061012565.us-east-2.elb.amazonaws.com:80

We’ll use this internal proxy address to test our routing to the get endpoint of the httpbin service. This endpoint simply echoes back the key elements of our request:

% curl $(glooctl proxy url)/get -i
HTTP/1.1 200 OK
date: Fri, 19 Nov 2021 19:04:05 GMT
content-type: application/json
content-length: 423
server: envoy
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 24

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "ac2440158914049ba8880519a5234c6c-2061012565.us-east-2.elb.amazonaws.com",
    "User-Agent": "curl/7.64.1",
    "X-Amzn-Trace-Id": "Root=1-6197f525-149d66cd7fcc3f683be38a0b",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "18.221.254.246",
  "url": "http://ac2440158914049ba8880519a5234c6c-2061012565.us-east-2.elb.amazonaws.com/get"
}

Good news! The standard internal Envoy proxy in our EKS cluster works as expected. But we’re just getting started.

7. Configure external Envoy on EC2

Now that we’ve confirmed the internal proxy is working properly, we can turn our attention to configuring the external Envoy proxy that will run in our out-of-cluster EC2 instance.

For more information about configuring an external Envoy proxy, you can check out the Gloo Edge documentation. However, note that this documentation describes how to establish the entire open source Gloo Edge microservice fleet using docker-compose. Our objective is to instead deploy only the Envoy proxy itself with docker-compose, while we depend on the standard EKS-deployed Gloo Edge to provide control plane services like gloo and discovery.  Additionally, this scenario uses Gloo Edge Enterprise instead of open source. If you need to change the Enterprise wrapper around Envoy to the open source wrapper, follow the one-line change as noted in the comments of docker-compose.yaml below.

First, we’ll use docker-compose tools to add the required directory structure on the EC2 instance to run our Envoy proxy. Issue this wget command to place the archive of configuration files on your EC2 instance:

wget https://github.com/solo-io/solo-blog/raw/main/non-kube-envoy/ec2-non-kube-envoy.tgz

Unpack the archive into a non-kube-envoy directory on your EC2 instance:

$ tar zxvf ./ec2-non-kube-envoy.tgz
./non-kube-envoy/
./non-kube-envoy/data/
./non-kube-envoy/data/envoy-config.yaml
./non-kube-envoy/docker-compose.yaml

Next, examine the key files, such as the following docker-compose.yaml file. This configuration shows that we are creating only the gateway-proxy outside of the cluster on the EC2 instance. As previously noted, you can also see that we are using the Gloo Edge Enterprise Envoy wrapper (gloo-ee-envoy-wrapper) rather than the open source wrapper.

version: '3'

services:

  gateway-proxy:
    # gloo-ee-envoy-wrapper is the Gloo Edge Enterprise base image.
    # For the open-source image, replace "gloo-ee-envoy-wrapper" with "gloo-envoy-wrapper" below.
    # You may also need to adjust the version number from 1.9.0 if you're using a non-1.9
    # Gloo Edge version.
    image: ${GLOO_REPO:-quay.io/solo-io}/gloo-ee-envoy-wrapper:${GLOO_VERSION:-1.9.0}
    entrypoint: ["envoy"]
    command: ["-c", "/config/envoy.yaml", "--disable-hot-restart"]
    volumes:
    - ./data/envoy-config.yaml:/config/envoy.yaml:ro
    ports:
    - "8080:8080"
    - "8443:8443"
    - "19000:19000"
    restart: always

Below is the envoy-config.yaml configuration for bootstrapping Envoy. For our scenario, we make a single modification to the standard bootstrap Envoy configuration. The Envoy xds_cluster identifies where the Envoy proxy retrieves its configuration. By default, this source is the gloo pod that represents the heart of the Gloo Edge control plane. We will instead use the load balancer endpoint that we created earlier by exposing the gloo service as a LoadBalancer service, rather than the default ClusterIP type. Be sure to replace that address: in the YAML with your gloo pod’s load balancer address that you found at the end of step 3.

node:
  cluster: gateway
  id: docker-compose-node
  metadata:
    # this line must match !
    role: "gloo-system~gateway-proxy"
static_resources:
  clusters:
  # The xds_cluster identifies where the Envoy proxy should retrieve its configuration.
  # By default, this is the gloo pod that represents the heart of the Gloo Edge control plane.
  - name: xds_cluster
    type: STRICT_DNS
    http2_protocol_options: {}
    connect_timeout: 5.000s
    load_assignment:
      cluster_name: xds_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                # address: gloo
                # Instead of using the standard in-cluster gloo pod to retrieve configuration,
                # we will use the load balancer endpoint established by exposing the gloo service
                # as a LoadBalancer service, rather than the default ClusterIP type.
                # Replace the address below with YOUR gloo service LB address.
                address: a7788fc21d90a4c3eb7e231344772a69-1799697922.us-east-2.elb.amazonaws.com
                port_value: 9977
dynamic_resources:
  ads_config:
    transport_api_version: V3
    api_type: GRPC
    grpc_services:
    - envoy_grpc: {cluster_name: xds_cluster}
  cds_config:
    ads: {}
    resource_api_version: V3
  lds_config:
    ads: {}
    resource_api_version: V3
admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 19000

8. Test with external Envoy proxy

Now that we have established the configuration for our EC2-deployed external Envoy proxy, let’s test it out.

First, from the directory where the docker-compose.yaml file is stored on EC2, use this command to bring up the proxy. Note that you’ll likely need sudo privileges for this to work properly.

[ec2-user@ip-172-31-44-67 non-kube-envoy]$ sudo /usr/local/bin/docker-compose up

If configuration is correct and Envoy can access the gloo service via the AWS load balancer, then you should see output that looks like the following:

[+] Running 1/0
 ⠿ Container non-kube-envoy-gateway-proxy-1 Created 0.0s
Attaching to non-kube-envoy-gateway-proxy-1
non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:08.340][1][info][main] [external/envoy/source/server/server.cc:338] initializing epoch 0 (base id=0, hot restart version=disabled)
non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:08.340][1][info][main] [external/envoy/source/server/server.cc:340] statically linked extensions:
... snip ...
non-kube-envoy-gateway-proxy-1  | [2021-11-30 23:18:13.923][1][info][upstream] [external/envoy/source/server/lds_api.cc:78] lds: add/update listener 'listener-::-8080'
non-kube-envoy-gateway-proxy-1  | [2021-11-30 23:18:13.925][1][info][config] [external/envoy/source/server/listener_manager_impl.cc:834] all dependencies initialized. starting workers

You may notice some benign warnings in the Envoy logs that you can safely ignore. First are a series of warnings about use of a deprecated http2_protocol_options option, like the one shown below. This is a benign issue in the 1.9.x version of Gloo Edge that will be cleaned up before support is removed from Envoy altogether.

non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:08.892][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.

Second, you are likely to notice some “connection termination” warnings in the Envoy logs occurring at approximately one-minute intervals once the proxy has spun up. These are also benign warnings due to an Envoy issue that has been cleaned up in Envoy 1.20.0 (October 2021) and will be incorporated into a post-1.9 release of Gloo Edge. Here is an example of this warning.

non-kube-envoy-gateway-proxy-1 | [2021-12-01 02:20:15.015][1][warning][config] [external/envoy/source/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination

Finally, let’s invoke some httpbin endpoints by using the Envoy proxy deployed on our EC2 instance. For example, you can curl the EC2 hostname provisioned by AWS from your local workstation. Note that you might need to configure your AWS security groups to allow remote access.

# Access the httpbin get endpoint via the out-of-cluster Envoy
% curl http://ec2-18-188-39-178.us-east-2.compute.amazonaws.com:8080/get -i
HTTP/1.1 200 OK
date: Fri, 19 Nov 2021 20:08:58 GMT
content-type: application/json
content-length: 378
server: envoy
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 24
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "ec2-18-188-39-178.us-east-2.compute.amazonaws.com",
    "User-Agent": "curl/7.64.1",
    "X-Amzn-Trace-Id": "Root=1-6198045a-5e8a01854b08f43d47ada43d",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "18.188.39.178",
  "url": "http://ec2-18-188-39-178.us-east-2.compute.amazonaws.com/get"
}

# Access the httpbin status endpoint via the out-of-cluster Envoy
% curl http://ec2-18-188-39-178.us-east-2.compute.amazonaws.com:8080/status/200 -i
HTTP/1.1 200 OK
date: Fri, 19 Nov 2021 20:12:31 GMT
content-type: text/html; charset=utf-8
content-length: 0
server: envoy
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 22

Learn more about deploying Envoy proxies outside Kubernetes with Gloo Edge

If you’ve made it this far, you’ve accomplished quite a bit. You have established a standard Gloo Edge Enterprise installation on EKS, wired up its gloo service for external access, configured an external service and routing rules to reach it, and then accessed all of that using an out-of-cluster Envoy proxy deployed on EC2. Congratulations, you’ve completed deploying Envoy proxies outside of Kubernetes using Gloo Edge!

All of the code used in this post is available on github.

For more information on Gloo Edge, check out the following resources.

Acknowledgments

Thank you to my colleagues Lawrence Gadban and Adam Sayah at solo.io for their valuable contributions to this post.