Canary Deployments with Gloo Function Gateway using Request Headers
This is the 2nd post in my 3 part series on doing Canary Releases with Solo.io Gloo.
- In part 1, Routing with Gloo Function Gateway, you learned how to setup Gloo, and using Gloo to set up some initial function level routing rules.
- In part 2, Canary Deployments with Gloo Function Gateway, you learned how to set up a conditional routing rule that routed requests to the new version of our service only when a request header was present with the correct value.
- In part 3, Canary Deployments with Gloo Function Gateway using Weighted Destinations, we used weighted destinations to route a percentage of request traffic to individual upstream services.
This post expands on the Function Routing with Gloo post to show you how to do a Canary release of a new version of a function. Gloo is a function gateway that gives users a number of benefits including sophisticated function level routing, and deep service discovery with introspection of OpenAPI (Swagger) definitions, gRPC reflection, Lambda discovery and more. This post will show a simple example of Gloo discovering 2 different deployments of a service, and setting up some routes. The route rules will use the presence of a request header x-canary:true to influence runtime routing to either version 1 or version 2 of our function. Then once we’re happy with our new version, we will update the route so all requests now go to version 2 of our service. All without changing or even redeploying our 2 services. But first, let’s set some context…
Background
Canary release is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody.
Danilo Sato Canary Release
The idea of a Canary release is that no matter how much testing you do on a new implementation, until you deploy it into your production environment you can’t be positive everything will work as expected. So having a way to release a new version into production concurrently with the existing version(s) with some way to route traffic can be helpful. Ideally, we’d like to route most traffic to existing, known to work version, and have a way for some (test) requests go to the new version. Once you’re feeling comfortable that your new version is working as you expect, then and only then, do you start routing most/all requests to the new version, and then eventually decommission the original service.
Being able to change request routes without needing to change or redeploy your code, I think, is very helpful in building confidence that your code is ready for production. That is, if you need to change your code or use a code based feature flag, then your exercising different code paths and/or changing deployed configuration settings. I feel its better if you can deploy your service, code and configurations, all ready for production, and use an external mechanism to manage request routing.
Gloo uses Envoy, which is a super high performance service proxy, to do the request routing. In this example, we’ll use a request header to influence the routing, though we could also use other variables like the IP range of the requestor to drive routing decisions. That is, if requests are coming from specific test machines we can route them to our new version. Lots more information on how Gloo and Envoy works can be found on the Solo.io website. On to the example…
This post assumes you’ve already run thru the Function Routing with Gloo post, and that you’ve already got a Kubernetes environment setup with Gloo. If not, please refer back to that post for setup instructions and the basics of VirtualServices and Routes with Gloo.
All of the Kubernetes manifests are located at https://github.com/scranton/gloo-canary-example. I’d suggest you clone that repo locally to make it easier to try these example yourself. All command examples assume you are in the top level directory of that repo.
Review
In the previous post, we had a single service petstore-v1, and we setup Gloo to route requests to its findPets REST function. Let’s test that its still working as expected. Remember we need to get Gloo’s proxy url by calling the glooctl proxy url command, and then we can make requests against that with the /findPets route that we previously setup. If still working correctly we should get 2 Pets back.
export PROXY_URL=$(glooctl proxy url)
curl ${PROXY_URL}/findPets[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]
Canary Routing
Now let’s deploy a version 2 of our service, and let’s setup a canary route for the findPets function. That is, by default we’ll route to version 1 of the function, and if there is a request header x-canary:true set, we’ll route that request to version 2 of our function.
Install and verify petstore version 2 example service
Let’s first deploy version 2 of our petstore service. This version has been modified to return 3 pets.
kubectl apply -f petstore-v2.yaml
This Github Sample is by scranton petstore-v2.yaml view raw
---# petstore-v2
apiVersion: v1
kind: Service
metadata:
name: petstore-v2
namespace: default
labels:
app: petstore-v2
spec:
type: ClusterIP
ports:
- name: http
port: 8080
targetPort: 8080
protocol: TCP
selector:
app: petstore-v2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: petstore-v2
namespace: default
labels:
app: petstore-v2
spec:
replicas: 1
selector:
matchLabels:
app: petstore-v2
template:
metadata:
labels:
app: petstore-v2
spec:
containers:
- name: petstore-v2
image: scottcranton/petstore:v2
ports:
- containerPort: 8080
Verify its setup right
kubectl get services --namespace default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 22h petstore-v1 ClusterIP 10.110.99.86 8080/TCP 33m petstore-v2 ClusterIP 10.109.91.120 8080/TCP 6s
Now let’s setup a port forward to see if it works. When we do a GET against /api/pets we should get back 3 pets.
kubectl port-forward services/petstore-v2 8080:8080
And in a different terminal, run the following to see if we get back 3 pets for version 2 of our service.
curl localhost:8080/api/pets[{"id":1,"name":"Dog","status":"v2"},{"id":2,"name":"Cat","status":"v2"},{"id":3,"name":"Parrot","status":"v2"}]
You should kill all port forwarding as we’ll use Gloo to proxy future tests.
Setup Canary Route
Let’s setup a new function route rule for petstore version 2 findPets function that depends on the presence of the x-canary:true request header.
glooctl add route \ --name coalmine \ --path-prefix /findPets \ --dest-name default-petstore-v2-8080 \ --rest-function-name findPets \ --header x-canary=true
Default routing should still go to petstore version 1, and return only 2 pets.
curl ${PROXY_URL}/findPets[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]
If we make a request with the x-canary:true set, it should route to petstore version 2, and return 3 pets.
curl -H "x-canary:true" ${PROXY_URL}/findPets[{"id":1,"name":"Dog","status":"v2"},{"id":2,"name":"Cat","status":"v2"},{"id":3,"name":"Parrot","status":"v2"}]
Just to verify, let’s set the header to a different value, e.g. x-canary:false to see that it routes to petstore v1
curl -H "x-canary:false" ${PROXY_URL}/findPets [{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]
Here’s the complete YAML for our coalmine virtual service that you could kubectl apply if you wanted to recreate.
This Github Sample is by scranton coalmine-virtual-service-part-2-header.yaml view raw
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: coalmine
namespace: gloo-system
spec:
displayName: coalmine
virtualHost:
domains:
- '*'
name: gloo-system.coalmine
routes:
- matcher:
headers:
- name: x-canary
regex: true
value: "true"
prefix: /findPets
routeAction:
single:
destinationSpec:
rest:
functionName: findPets
parameters: {}
upstream:
name: default-petstore-v2-8080
namespace: gloo-system
- matcher:
prefix: /findPetWithId
routeAction:
single:
destinationSpec:
rest:
functionName: findPetById
parameters:
headers:
:path: /findPetWithId/{id}
upstream:
name: default-petstore-v1-8080
namespace: gloo-system
- matcher:
prefix: /findPets
routeAction:
single:
destinationSpec:
rest:
functionName: findPets
parameters: {}
upstream:
name: default-petstore-v1-8080
namespace: gloo-system
- matcher:
prefix: /petstore
routeAction:
single:
upstream:
name: default-petstore-v1-8080
namespace: gloo-system
routePlugins:
prefixRewrite:
prefixRewrite: /api/pets
The part of the virtual service manifest that is specifying the header based routing is highlighted as follows.
- matcher:
headers:
- name: x-canary
regex: true
value: "true"
prefix: /findPets
routeAction:
Make version 2 the default for all requests
Once we’re feeling good about version 2 of our function, we can make the default call to /findPets go to version 2. Note that will Gloo as your function gateway, you do not have to route all function requests to version 2 of the petstore service. In this example, we’re only routing requests for the findPets function to version 2. All other requests are going to version 1 of petstore. This partial routing may not always work for all services; this post is showing that Gloo makes this level of granularity possible when it helps you more fine tune your application upgrading decisions. For example, this may make sense if you want to patch a critical bug but are not ready to role out other breaking changes in a new service version.
The easiest way to make change the routing rules to route all requests to version 2 findPets is by applying a YAML file. You can use the glooctl command line tool to add and remove routes, but it takes several calls.
This Github Sample is by scranton coalmine-virtual-service-part-2-v2.yaml view raw
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: coalmine
namespace: gloo-system
spec:
displayName: coalmine
virtualHost:
domains:
- '*'
name: gloo-system.coalmine
routes:
- matcher:
prefix: /findPets
routeAction:
single:
destinationSpec:
rest:
functionName: findPets
parameters: {}
upstream:
name: default-petstore-v2-8080
namespace: gloo-system
- matcher:
prefix: /findPetWithId
routeAction:
single:
destinationSpec:
rest:
functionName: findPetById
parameters:
headers:
:path: /findPetWithId/{id}
upstream:
name: default-petstore-v1-8080
namespace: gloo-system
- matcher:
prefix: /petstore
routeAction:
single:
upstream:
name: default-petstore-v1-8080
namespace: gloo-system
routePlugins:
prefixRewrite:
prefixRewrite: /api/pets
Summary
This post has shown you have to leverage the Gloo function gateway to do a Canary Release of a new version of a function, and allow you to do very granular function level routing to validate your new function is working correctly. Then it showed changing routing rules so all traffic goes to the new version. All without redeploying either of the 2 service implementations. In this post we used the presence of a request header to influence function routing; we could also have done routing based on IP range of incoming request or other variables. This hopefully shows you the power and flexibility that Gloo function gateway can provide you in your journey to microservices and service mesh.
