Extending Gloo Edge Made Easy

Introduction

Gloo Edge is a cloud-native API Gateway and Ingress Controller built on Envoy Proxy to facilitate and secure application traffic at the edge.

When you implement Gloo Edge Enterprise, you can take advantage of a large number of features: you can easily transform your requests (headers, body, …), secure your applications (OIDC, JWT, OPA, WAF, …) and to perform many other operations.

But there are also cases where you want to implement your own custom logic. For example, you may want Gloo Edge to Authenticate a request and then use your own internal service to Authorize it.

When we speak about extending Gloo Edge (and Envoy in general), we quickly think about WASM (Web Assembly). WASM is definitely the future of Envoy and will become the right way to implement custom workflows. But, unfortunately, it’s not quite ready yet for production use.

I encourage you to take a look at the Blog post to understand the current state of WASM in Envoy.

So, what if you want to extend Gloo Edge today ?

We’ve provided a way to do that by developing an Extauth plugin in Go. It allows you to use the full power of Go to implement your custom logic. But you need to recompile your plugin for each Gloo Edge version.

So, we decided to provide a new option in Gloo Edge 1.6 that we call Passthrough Auth. You can now run your own service independently from Gloo Edge (as soon as it implements Envoy’s authorization service API) which will be called by Gloo Edge’s external auth service through gRPC.

It does require an additional network hop, but it gives you a lot of flexibility (and you don’t need to recompile it for each Gloo Edge version).

We’ve published a nice comparison in the Gloo Edge documentation here.

Passthrough Auth can be used to implement your custom Authorization after Gloo Edge Authenticated the request using OAuth (for example), but it can also be used  for any complex request manipulation you’d like to perform. So, the possibilities are limitless!

In this Blog post, we will implement Passthrough Auth to send an API call to ip-api.com to enrich the requests with additional headers regarding the location of the user who sent the request. Special thanks to Lior Kuperiu from DAZN who shared this use case with me.

Prepare the environment

I’ve deployed a Kubernetes cluster (EKS) on AWS, so I can now deploy the latest Gloo Edge Enterprise version using the commands below:

cat > values.yaml <<EOF
gloo:
  gatewayProxies:
    gatewayProxy:
      service:
        extraAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
EOF

glooctl install gateway enterprise --version 1.6.2 --values values.yaml --license-key $LICENSE_KEY

Note that I’ve added an annotation on the gateway-proxy service to enable Proxy Protocol support on the Classic Load Balancer that will be provisioned.

Then, I use the following yaml to deploy the httpbin application which will allow me to see the headers it receives.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-deployment-1
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin-1
  template:
    metadata:
      labels:
        app: httpbin-1
    spec:
      containers:
        - name: httpbin-1
          image: kennethreitz/httpbin
          ports:
            - name: http-port
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-service-1
  namespace: default
spec:
  ports:
    - name: http-port
      port: 80
      targetPort: http-port
      protocol: TCP
  selector:
    app: httpbin-1

After that, I can create a Gloo Edge VirtualService to expose the application:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - "*"
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-service-1-80
            namespace: gloo-system

Finally, I need to patch the Gloo Edge Gateway to use the remote address passed by the AWS Load Balancer:

cat > patch.yaml <<EOF
spec:
  useProxyProto: true
  httpGateway:
    options:
      httpConnectionManagerSettings:
        useRemoteAddress: true
EOF

kubectl patch -n gloo-system gateway gateway-proxy --type merge --patch "$(cat patch.yaml)"

I can use `curl` to check what headers are received by the application:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "a3687af03f1544befb4ed987ac6ff732-1862501833.eu-west-1.elb.amazonaws.com", 
    "User-Agent": "curl/7.64.1", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000", 
    "X-Envoy-External-Address": "86.245.228.182"
  }, 
  "origin": "86.245.228.182", 
  "url": "http://a3687af03f1544befb4ed987ac6ff732-1862501833.eu-west-1.elb.amazonaws.com/get"
}

As expected, I can see my IP address.

Build and deploy the plugin

We provide a template in the Gloo Edge repo to help you develop your own Plugin very easily. It’s available here.

Here is the auth.go file I used to enrich the request with geo headers:

package v3

import (
	"context"
	"encoding/json"
	"io/ioutil"
	"log"
	"net/http"
	"reflect"
	"strconv"

	envoy_api_v3_core "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
	"github.com/golang/protobuf/ptypes/wrappers"

	envoy_service_auth_v3 "github.com/envoyproxy/go-control-plane/envoy/service/auth/v3"
	"google.golang.org/genproto/googleapis/rpc/code"
	"google.golang.org/genproto/googleapis/rpc/status"
)

type server struct {
}

type Geo struct {
	Status string `json:"status"`
	Country string `json:"country"`
	CountryCode string `json:"countryCode"`
	Region string `json:"region"`
	City string `json:"city"`
	Zip string `json:"zip"`
	Lat float64 `json:"lat"`
	Lon float64 `json:"lon"`
	Timezone string `json:"timezone"`
	Isp string `json:"isp"`
	Org string `json:"org"`
	As string `json:"as"`
	Query string `json:"query"`
}

var _ envoy_service_auth_v3.AuthorizationServer = &server{}

func New() envoy_service_auth_v3.AuthorizationServer {
	return &server{}
}

func StatusOK() (*envoy_service_auth_v3.CheckResponse, error) {
	return &envoy_service_auth_v3.CheckResponse{
		HttpResponse: &envoy_service_auth_v3.CheckResponse_OkResponse{
			OkResponse: &envoy_service_auth_v3.OkHttpResponse{},
		},
		Status: &status.Status{
			Code: int32(code.Code_OK),
		},
	}, nil
}

func (s *server) Check(
	ctx context.Context,
	req *envoy_service_auth_v3.CheckRequest) (*envoy_service_auth_v3.CheckResponse, error) {
	
	log.Println(req.Attributes.Request.Http.Headers)
	resp, err := http.Get("http://ip-api.com/json/" + req.Attributes.Request.Http.Headers["x-forwarded-for"] + "?fields=status,country,countryCode,region,city,zip,lat,lon,timezone,isp,org,as,query")
	if err != nil {
		return StatusOK()
	} else {
		body, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			return StatusOK()
		}

	    var geo Geo
		err = json.Unmarshal(body, &geo)
		if err != nil {
			return StatusOK()
		}

		v := reflect.ValueOf(geo)
    	typeOfS := v.Type()
    
		headers := []*envoy_api_v3_core.HeaderValueOption{}
		for i := 0; i< v.NumField(); i++ {
			key := typeOfS.Field(i).Name
			value := ""
			if(key == "Lat" || key == "Lon") {
				value = strconv.FormatFloat(v.Field(i).Interface().(float64), 'f', 6, 64)
			} else {
				value = v.Field(i).Interface().(string)
			}
			headers = append(headers, &envoy_api_v3_core.HeaderValueOption{
				Append: &wrappers.BoolValue{Value: false},
				Header: &envoy_api_v3_core.HeaderValue{
					Key:   "geo-" + key,
					Value: value,
				},
			})
		}

		return &envoy_service_auth_v3.CheckResponse{
			HttpResponse: &envoy_service_auth_v3.CheckResponse_OkResponse{
				OkResponse: &envoy_service_auth_v3.OkHttpResponse{
					Headers: headers,
				},
			},
			Status: &status.Status{
				Code: int32(code.Code_OK),
			},
		}, nil
	}
}

I won’t explain the code in details (and I’m sure an experienced Go developer would have done it in a better way), but I want to highlight the fact that you can access all the information about the requests (headers, …) and you can modify the requests the way you want. It’s really powerful and easy.

As you can see, it gets the client IP address from the x-forwarded-for header and sends an API request to ip-api.com to get geo information about the IP address. Then, it creates new headers based on the information returned by the API call.

If any error occurs (I can’t get the IP address, I can’t send the API call, …), I just return an OK Response and the request will continue without any modification.

I must compile the server binary:

make server

I can now build my Docker image using the following command:

docker build -t djannot/passthrough-grpc-service-example:v0.1 .

And I can then push it:

docker push djannot/passthrough-grpc-service-example:v0.1

After that, I can deploy it on my Kubernetes cluster using the following yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: extauth-grpcservice
spec:
  selector:
    matchLabels:
      app: grpc-extauth
  replicas: 1
  template:
    metadata:
      labels:
        app: grpc-extauth
    spec:
      containers:
        - name: grpc-extauth
          image: djannot/passthrough-grpc-service-example:v0.1
          imagePullPolicy: Always
          ports:
            - containerPort: 9001
---
apiVersion: v1
kind: Service
metadata:
  name: example-grpc-auth-service
  labels:
      app: grpc-extauth
spec:
  ports:
  - port: 9001
    protocol: TCP
  selector:
      app: grpc-extauth

Finally, I need to create an AuthConfig object to reference my Plugin:

apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: passthrough-auth
  namespace: gloo-system
spec:
  configs:
  - passThroughAuth:
      grpc:
        address: example-grpc-auth-service.default.svc.cluster.local:9001
        connectionTimeout: 3s

And I also need to update my VirtualService to use this AuthConfig:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - "*"
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-service-1-80
            namespace: gloo-system
    options:
      extauth:
        configRef:
          name: passthrough-auth
          namespace: gloo-system

Test the plugin

I can use `curl` again to check what headers are received by the application:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Geo-As": "AS3215 Orange S.A.", 
    "Geo-City": "Argenteuil", 
    "Geo-Country": "France", 
    "Geo-Countrycode": "FR", 
    "Geo-Isp": "Orange", 
    "Geo-Lat": "48.948400", 
    "Geo-Lon": "2.251300", 
    "Geo-Org": "", 
    "Geo-Query": "90.90.7.0", 
    "Geo-Region": "IDF", 
    "Geo-Status": "success", 
    "Geo-Timezone": "Europe/Paris", 
    "Geo-Zip": "95100", 
    "Host": "a3687af03f1544befb4ed987ac6ff732-1862501833.eu-west-1.elb.amazonaws.com", 
    "User-Agent": "curl/7.64.1", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000", 
    "X-Envoy-External-Address": "90.90.7.0"
  }, 
  "origin": "90.90.7.0", 
  "url": "http://a3687af03f1544befb4ed987ac6ff732-1862501833.eu-west-1.elb.amazonaws.com/get"
}

As expected, I now have new headers corresponding to the geo information related to the my IP address.

Very simple, no ?