Deploying Envoy proxies outside Kubernetes with Gloo Edge
Some things should always be separated, shouldn’t they? Like oil and water. Wheat and chaff. Church and state. And how about deploying Envoy proxies and their control planes, like Gloo Edge?
By default, you might think of them as living together. In the Gloo Edge documentation, the default scenario deploys an Envoy proxy and its control plane to one Kubernetes cluster, which is how many users deploy both in development and production.
But there’s a possibility that you will need to deploy the Envoy proxies and their control planes differently. Perhaps you’d like to safely ensconce your control plane behind a firewall in its Kubernetes cluster. Maybe you’d like to deploy the Envoy proxies of your data plane separately, such as in a DMZ (demilitarized zone) on virtual machines. Separating deployments is a common request from enterprise users, especially those with large fleets of legacy assets that existed before Kubernetes was on anyone’s radar.
Can Gloo Edge do that? Absolutely!
In this blog post, we’ll walk through a separate-deployment scenario in which Gloo Edge allows you to deploy Envoy proxies outside of Kubernetes. The outlined steps show you how to first deploy the control plane to an Amazon Elastic Kubernetes Service (EKS) cluster, and then deploy the data plane to a non-Kubernetes Amazon EC2 virtual machine.
An architectural foundation for deploying Envoy proxies with Gloo Edge
Gloo Edge is built with a cloud-native architecture. This means that it’s deployed as a fleet of microservices, commonly within a single Kubernetes cluster. Typically, the Envoy proxies that serve as the data plane exist within the same cluster as the Edge control plane components that collaborate to dynamically serve configuration to Envoy. To learn more about the various control plane components, see the Gloo Edge documentation.
The key interface when deploying a managed Envoy proxy outside a Kubernetes cluster is the Envoy-to-Gloo connection. In almost all enterprise deployments, Envoy depends on a management server that generates configuration dynamically as policies and components in the cluster change. That management server adheres to an Envoy protocol called xDS (all Discovery Services). The gloo
service acts as the xDS server to one or more Envoy proxies.
A default Gloo Edge installation implicitly assumes that the Envoy proxy, the gateway-proxy
service, is deployed in the same cluster as its xDS server, the gloo
service. Consequently, we’ll ensure for our purposes that the gloo
service is established as a LoadBalancer
service instead of its default ClusterIP
type. That will allow the Envoy gateway-proxy
to talk to its gloo
xDS server in order to receive its configuration.
Deploying Envoy proxies step by step
To get started, we’ll establish a Kubernetes-deployed Gloo Edge control plane that manages a non-Kubernetes Envoy proxy. These steps use an Amazon EKS cluster to host the control plane and an out-of-cluster EC2 instance to host the standalone Envoy proxy. We’ll also pay special attention to the Gloo Edge bits of the configuration process.
Here’s a summary of the steps in the process:
- Provision AWS components, specifically an EKS cluster and an EC2 instance.
- Configure Gloo Edge Enterprise on the EKS cluster.
- Ensure
gloo
xDS server is available outside the cluster. - Configure an external target service for testing.
- Configure a Gloo Edge VirtualService.
- Test with the internal Envoy proxy.
- Configure external Envoy on the EC2 instance.
- Test with the external Envoy proxy.
1. Provision AWS components
The general principles outlined in this post should work with any cloud provider. But in this case, we’ll work with AWS, using both an EKS cluster and a standalone EC2 instance. This basic approach is also known to work with Google GKE clusters.
Establish an EKS cluster
First, we’ll assume that you have access to an EKS cluster, or sufficient AWS permissions to create a cluster. If you don’t have the proper access, check out this blog post to get started with Gloo Edge on EKS.
If you’re creating a cluster, you can use the AWS console or the eksctl
CLI. We used the eksctl
CLI utility as a simpler alternative to the console. Try this command with the eksctl
CLI to create a modest demo cluster with two m5.large instances for this scenario:
eksctl create cluster --name non-kube-envoy --region us-east-2
This command creates a cluster in the Ohio (us-east-2) region. If you don’t specify a region, eksctl
defaults to the Oregon (us-west-2) region. A full AWS region list with corresponding codes is available here.
Configure an EC2 instance
Second, we created a standalone EC2 instance outside the EKS cluster to host our non-Kubernetes Envoy proxy. This was an m5.large instance in the us-east-2 region. If you don’t have experience provisioning a standalone EC2, you might find the articles here and here helpful.
After you establish SSH (Secure Shell) access to your EC2 instance, you will need to install docker tools, including docker-compose. We used the following sequence on our EC2 instance:
$ sudo yum install docker $ sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose $ sudo chmod +x /usr/local/bin/docker-compose
# Confirm installation $ docker-compose version Docker Compose version v2.0.1
At this point, you should have an EKS cluster that allows access from remote addresses. You should also have an EC2 instance that contains basic docker tools and allows external SSH and HTTP access to facilitate testing.
2. Configure Gloo Edge Enterprise
We will use the Enterprise edition of Gloo Edge in these steps. The open source edition is also sufficient, with only minor configuration tweaks. However, note that in the forthcoming blog post for this scenario, which shows you how to extend the data plane deployment to include rate limiting and external auth servers, a Gloo Edge Enterprise license is required. You can request an Enterprise license key here.
If you haven’t already installed a Gloo Edge instance on a cluster, follow this guide to install the glooctl
CLI and the Gloo Edge Enterprise server in a Kubernetes cluster. Note that we used Gloo Edge Enterprise version 1.9.2 for this scenario.
3. Ensure gloo xDS is available outside your cluster
As mentioned earlier, Gloo Edge typically assumes that the Envoy proxy service is deployed in the same cluster as its xDS server. That is not the case in this scenario, in which we are deploying Envoy outside of the Kubernetes cluster. Consequently, we need to change the default service type of the gloo
service, since Envoy (gateway-proxy
) needs to connect with it directly to receive its configuration. To achieve that, we can patch the gloo
service to change its service type from the default ClusterIP
to LoadBalancer
.
Once that change takes effect, AWS creates a CLB (Classic Load Balancer) instance that we can direct our Envoy proxy to use. Though we use the default CLB in this exercise, we also tested this successfully with an AWS NLB (Network Load Balancer). NLB is the more modern technology, and multiple Gloo Edge users have successfully configured it in production deployments.
kubectl patch svc gloo -n gloo-system -p '{"spec": {"type": "LoadBalancer"}}'
You should see a response like this:
service/gloo patched
To confirm that this patch worked as expected, first inspect the service type:
kubectl get svc gloo -n gloo-system -o=jsonpath='{.spec.type}'
In the ouput, verify that gloo
is now a LoadBalancer
service.
Next, verify that AWS has provisioned a proper elb
hostname as the gloo
external hostname:
kubectl get svc gloo -n gloo-system -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
You should see an AWS load balancer hostname, for example:
a7788fc21d90a4c3eb7e231344772a69-1799697922.us-east-2.elb.amazonaws.com
4. Configure the external target service
In this deployment pattern, the target service must either exist outside of a Kubernetes cluster or be routed to through an external endpoint. Keep in mind that the Envoy proxy that will handle this request is on an external VM, so it has no direct access to Kubernetes Services
. We will simulate that scenario by creating an Upstream for the public httpbin service. This offers the added benefit of allowing us to test a number of different HTTP request types using a single service.
apiVersion: gloo.solo.io/v1 kind: Upstream metadata: name: httpbin-us namespace: gloo-system spec: static: hosts: - addr: httpbin.org port: 80
Add this Upstream to your Gloo Edge configuration:
kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/non-kube-envoy/httpbin-us.yaml
5. Configure the Edge virtual service
We will configure an Edge VirtualService to specify the routing rules that will direct requests to the external httpbin service.
apiVersion: gateway.solo.io/v1 kind: VirtualService metadata: name: httpbin-vs namespace: gloo-system spec: virtualHost: domains: - '*' routes: - matchers: - prefix: / routeAction: single: upstream: name: httpbin-us namespace: gloo-system
Add this VirtualService to your Gloo Edge configuration:
kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/non-kube-envoy/httpbin-vs-ext.yaml
6. Test with internal Envoy proxy
Now that we defined both a VirtualService and an Upstream for an external service, we can confirm that everything is working by invoking the standard Envoy proxy deployed by Gloo Edge. (Note that if we were planning to take a non-Kubernetes Envoy deployment to production, we would disable this standard internal proxy.)
Once we established the valid Upstream and VirtualService, EKS automatically created an AWS Classic Load Balancer (CLB) in front of the internal Envoy proxy. You can find the hostname and URL associated with that proxy by using this glooctl
command:
% glooctl proxy url http://ac2440158914049ba8880519a5234c6c-2061012565.us-east-2.elb.amazonaws.com:80
We’ll use this internal proxy address to test our routing to the get
endpoint of the httpbin service. This endpoint simply echoes back the key elements of our request:
% curl $(glooctl proxy url)/get -i HTTP/1.1 200 OK date: Fri, 19 Nov 2021 19:04:05 GMT content-type: application/json content-length: 423 server: envoy access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 24 { "args": {}, "headers": { "Accept": "*/*", "Host": "ac2440158914049ba8880519a5234c6c-2061012565.us-east-2.elb.amazonaws.com", "User-Agent": "curl/7.64.1", "X-Amzn-Trace-Id": "Root=1-6197f525-149d66cd7fcc3f683be38a0b", "X-Envoy-Expected-Rq-Timeout-Ms": "15000" }, "origin": "18.221.254.246", "url": "http://ac2440158914049ba8880519a5234c6c-2061012565.us-east-2.elb.amazonaws.com/get" }
Good news! The standard internal Envoy proxy in our EKS cluster works as expected. But we’re just getting started.
7. Configure external Envoy on EC2
Now that we’ve confirmed the internal proxy is working properly, we can turn our attention to configuring the external Envoy proxy that will run in our out-of-cluster EC2 instance.
For more information about configuring an external Envoy proxy, you can check out the Gloo Edge documentation. However, note that this documentation describes how to establish the entire open source Gloo Edge microservice fleet using docker-compose. Our objective is to instead deploy only the Envoy proxy itself with docker-compose, while we depend on the standard EKS-deployed Gloo Edge to provide control plane services like gloo
and discovery
. Additionally, this scenario uses Gloo Edge Enterprise instead of open source. If you need to change the Enterprise wrapper around Envoy to the open source wrapper, follow the one-line change as noted in the comments of docker-compose.yaml
below.
First, we’ll use docker-compose tools to add the required directory structure on the EC2 instance to run our Envoy proxy. Issue this wget
command to place the archive of configuration files on your EC2 instance:
wget https://github.com/solo-io/solo-blog/raw/main/non-kube-envoy/ec2-non-kube-envoy.tgz
Unpack the archive into a non-kube-envoy
directory on your EC2 instance:
$ tar zxvf ./ec2-non-kube-envoy.tgz ./non-kube-envoy/ ./non-kube-envoy/data/ ./non-kube-envoy/data/envoy-config.yaml ./non-kube-envoy/docker-compose.yaml
Next, examine the key files, such as the following docker-compose.yaml
file. This configuration shows that we are creating only the gateway-proxy
outside of the cluster on the EC2 instance. As previously noted, you can also see that we are using the Gloo Edge Enterprise Envoy wrapper (gloo-ee-envoy-wrapper
) rather than the open source wrapper.
version: '3' services: gateway-proxy: # gloo-ee-envoy-wrapper is the Gloo Edge Enterprise base image. # For the open-source image, replace "gloo-ee-envoy-wrapper" with "gloo-envoy-wrapper" below. # You may also need to adjust the version number from 1.9.0 if you're using a non-1.9 # Gloo Edge version. image: ${GLOO_REPO:-quay.io/solo-io}/gloo-ee-envoy-wrapper:${GLOO_VERSION:-1.9.0} entrypoint: ["envoy"] command: ["-c", "/config/envoy.yaml", "--disable-hot-restart"] volumes: - ./data/envoy-config.yaml:/config/envoy.yaml:ro ports: - "8080:8080" - "8443:8443" - "19000:19000" restart: always
Below is the envoy-config.yaml
configuration for bootstrapping Envoy. For our scenario, we make a single modification to the standard bootstrap Envoy configuration. The Envoy xds_cluster
identifies where the Envoy proxy retrieves its configuration. By default, this source is the gloo
pod that represents the heart of the Gloo Edge control plane. We will instead use the load balancer endpoint that we created earlier by exposing the gloo
service as a LoadBalancer
service, rather than the default ClusterIP
type. Be sure to replace that address:
in the YAML with your gloo
pod’s load balancer address that you found at the end of step 3.
node: cluster: gateway id: docker-compose-node metadata: # this line must match ! role: "gloo-system~gateway-proxy" static_resources: clusters: # The xds_cluster identifies where the Envoy proxy should retrieve its configuration. # By default, this is the gloo pod that represents the heart of the Gloo Edge control plane. - name: xds_cluster type: STRICT_DNS http2_protocol_options: {} connect_timeout: 5.000s load_assignment: cluster_name: xds_cluster endpoints: - lb_endpoints: - endpoint: address: socket_address: # address: gloo # Instead of using the standard in-cluster gloo pod to retrieve configuration, # we will use the load balancer endpoint established by exposing the gloo service # as a LoadBalancer service, rather than the default ClusterIP type. # Replace the address below with YOUR gloo service LB address. address: a7788fc21d90a4c3eb7e231344772a69-1799697922.us-east-2.elb.amazonaws.com port_value: 9977 dynamic_resources: ads_config: transport_api_version: V3 api_type: GRPC grpc_services: - envoy_grpc: {cluster_name: xds_cluster} cds_config: ads: {} resource_api_version: V3 lds_config: ads: {} resource_api_version: V3 admin: access_log_path: /dev/null address: socket_address: address: 0.0.0.0 port_value: 19000
8. Test with external Envoy proxy
Now that we have established the configuration for our EC2-deployed external Envoy proxy, let’s test it out.
First, from the directory where the docker-compose.yaml
file is stored on EC2, use this command to bring up the proxy. Note that you’ll likely need sudo
privileges for this to work properly.
[ec2-user@ip-172-31-44-67 non-kube-envoy]$ sudo /usr/local/bin/docker-compose up
If configuration is correct and Envoy can access the gloo
service via the AWS load balancer, then you should see output that looks like the following:
[+] Running 1/0 ⠿ Container non-kube-envoy-gateway-proxy-1 Created 0.0s Attaching to non-kube-envoy-gateway-proxy-1 non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:08.340][1][info][main] [external/envoy/source/server/server.cc:338] initializing epoch 0 (base id=0, hot restart version=disabled) non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:08.340][1][info][main] [external/envoy/source/server/server.cc:340] statically linked extensions: ... snip ... non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:13.923][1][info][upstream] [external/envoy/source/server/lds_api.cc:78] lds: add/update listener 'listener-::-8080' non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:13.925][1][info][config] [external/envoy/source/server/listener_manager_impl.cc:834] all dependencies initialized. starting workers
You may notice some benign warnings in the Envoy logs that you can safely ignore. First are a series of warnings about use of a deprecated http2_protocol_options
option, like the one shown below. This is a benign issue in the 1.9.x version of Gloo Edge that will be cleaned up before support is removed from Envoy altogether.
non-kube-envoy-gateway-proxy-1 | [2021-11-30 23:18:08.892][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
Second, you are likely to notice some “connection termination” warnings in the Envoy logs occurring at approximately one-minute intervals once the proxy has spun up. These are also benign warnings due to an Envoy issue that has been cleaned up in Envoy 1.20.0 (October 2021) and will be incorporated into a post-1.9 release of Gloo Edge. Here is an example of this warning.
non-kube-envoy-gateway-proxy-1 | [2021-12-01 02:20:15.015][1][warning][config] [external/envoy/source/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
Finally, let’s invoke some httpbin endpoints by using the Envoy proxy deployed on our EC2 instance. For example, you can curl the EC2 hostname provisioned by AWS from your local workstation. Note that you might need to configure your AWS security groups to allow remote access.
# Access the httpbin get endpoint via the out-of-cluster Envoy % curl http://ec2-18-188-39-178.us-east-2.compute.amazonaws.com:8080/get -i HTTP/1.1 200 OK date: Fri, 19 Nov 2021 20:08:58 GMT content-type: application/json content-length: 378 server: envoy access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 24 { "args": {}, "headers": { "Accept": "*/*", "Host": "ec2-18-188-39-178.us-east-2.compute.amazonaws.com", "User-Agent": "curl/7.64.1", "X-Amzn-Trace-Id": "Root=1-6198045a-5e8a01854b08f43d47ada43d", "X-Envoy-Expected-Rq-Timeout-Ms": "15000" }, "origin": "18.188.39.178", "url": "http://ec2-18-188-39-178.us-east-2.compute.amazonaws.com/get" } # Access the httpbin status endpoint via the out-of-cluster Envoy % curl http://ec2-18-188-39-178.us-east-2.compute.amazonaws.com:8080/status/200 -i HTTP/1.1 200 OK date: Fri, 19 Nov 2021 20:12:31 GMT content-type: text/html; charset=utf-8 content-length: 0 server: envoy access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 22
Learn more about deploying Envoy proxies outside Kubernetes with Gloo Edge
If you’ve made it this far, you’ve accomplished quite a bit. You have established a standard Gloo Edge Enterprise installation on EKS, wired up its gloo
service for external access, configured an external service and routing rules to reach it, and then accessed all of that using an out-of-cluster Envoy proxy deployed on EC2. Congratulations, you’ve completed deploying Envoy proxies outside of Kubernetes using Gloo Edge!
All of the code used in this post is available on github.
- Explore the documentation for Gloo Edge.
- Request a live demo or trial for Gloo Edge Enterprise.
- See video content on the solo.io YouTube channel.
- Questions? Join the Solo.io Slack community and check out the #gloo-edge channel.
Acknowledgments
Thank you to my colleagues Lawrence Gadban and Adam Sayah at solo.io for their valuable contributions to this post.