How to turn application logs into useful metrics with Gloo Edge and Grafana
Monitoring your applications is a requirement in the cloud native world, but having access to useful metrics is often a difficult task. For some, the phase when metrics become most useful begins at the end of the development lifecycle, mostly during the day-2 operations (post-deployment). At that time, your operations teams can leverage the metrics to monitor your applications and to have a better understanding of what is happening.
Yet development teams work at the very beginning of the development lifecycle. To have good and meaningful metrics, some iterations and feedback are required. In reality, developers care most about features, tests, and… logs traces! Developers dream of having good log traces because then they can properly debug their applications.
So, how can you improve this experience?
Turning application logs into useful metrics
Promtail is an agent which ships the contents of local logs to a log aggregator. The magic happens when, besides shipping logs, it can also parse them. That allows you to configure transformations which can be translated into meaningful metrics.
To have the picture, think of a log trace. For example:
[ERROR] Error trying to connect to the endpoint. Connection reset.
You could parse the log traces which contains the expression [ERROR] and create a counter metric out of it to be displayed in your favorite monitoring system, such as Grafana and Prometheus.
A typical developer scenario for logs and metrics
One of our customers came with a question. They had configured OPA (Open Policy Agent) for their passthrough server as described in the documentation. Unfortunately, the OPA service was not running when they deployed the Gloo Edge resource (AuthConfig).
The event threw an error in the ExtAuth pod logs:
gloo-system extauth-67b888f686-2s7cr extauth {"level":"error","ts":" [...] ",
"caller":"config/generator.go:114","msg":"Errors encountered while processing new server configuration",
"version":"1.9.7","error":"1 error occurred:\n\t* failed to get auth service for auth config
with id [gloo-system.passthrough-auth]; this configuration
will be ignored: failed to create grpc client manager: context deadline exceeded\n\n",
"stacktrace":" [...] "}
The customer was quite interested in having everything covered by metrics. But when they discovered the issue, there were no metrics implemented to catch this scenario. The process to fix this issue involved opening a ticket and waiting for the engineering team to work on it. This delay could be easily tackled with the technique you will see in the next section.
Using Gloo Edge to build metrics from logs
The goal of this workshop is to catch ExtAuth (one of the Gloo Edge components) log traces so that they can be converted into metrics and exposed by a Grafana dashboard. Let’s walk through a common scenario and solution.
Your architecture will look like this:
Your Admin user is in charge of applying GlooEdge resources, while your Devops team is in charge of monitoring the infrastructure. You know that the error log is thrown when you apply an AuthConfig resource. In that resource, you specify the address to the OPA resource, which is not reachable.
Setup the environment
For this workshop you need to have:
- A Kubernetes cluster
- Helm
- A GlooEdge Enteprise license key
NOTE: Occasionally the button to “copy” might not work in some browsers. If so, please, select the text and copy.
Install Gloo Edge with a LICENSE_KEY:
export LICENSE_KEY=<here-your-license> helm repo add glooe https://storage.googleapis.com/gloo-ee-helm helm repo update helm upgrade --install gloo glooe/gloo-ee --namespace gloo-system \ --create-namespace --version 1.10.1 --set-string license_key=$LICENSE_KEY --devel
First, let’s discover the error. Create an application (i.e. httpbin):
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.12.0/samples/httpbin/httpbin.yaml
And expose it through Gloo Edge:
kubectl apply -f - <<EOF apiVersion: gateway.solo.io/v1 kind: VirtualService metadata: name: demo namespace: gloo-system spec: virtualHost: domains: - "*" routes: - matchers: - prefix: / options: extauth: configRef: name: passthrough-auth namespace: gloo-system routeAction: single: upstream: name: default-httpbin-80 namespace: gloo-system EOF
Create a file for the AuthConfig manifest so that you can then apply it and delete it easily:
cat << EOF > authconfig.yaml apiVersion: enterprise.gloo.solo.io/v1 kind: AuthConfig metadata: name: passthrough-auth namespace: gloo-system spec: configs: - passThroughAuth: grpc: address: ext-authz.default.svc.cluster.local:9000 connectionTimeout: 30s EOF
Now, apply it:
kubectl apply -f authconfig.yaml
Now since the service opa.opa.svc.cluster.local:9191 does not exist, an error is thrown in the ExtAuth pod.
Let’s verify it:
kubectl logs -l gloo=extauth -n gloo-system | grep "failed to create grpc client manager"
And you will see following error:
NOTE: If you do not see the error, wait a bit and try again. Propagating the events can take a bit of time.
gloo-system extauth-67b888f686-2s7cr extauth {"level":"error","ts":" [...] ",
"caller":"config/generator.go:114",
"msg":"Errors encountered while processing new server configuration",
"version":"1.9.7","error":"1 error occurred:\n\t* failed to get auth
service for auth config with id [gloo-system.passthrough-auth]; this configuration
will be ignored: failed to create grpc client manager: context deadline exceeded\n\n",
"stacktrace":" [...] "}
Now, you need to catch this error with Promtail, parse it as metric, and let Prometheus scrape the metric.
Let’s delete first the AuthConfig which triggered the error, to have a clean scenario:
kubectl delete -f authconfig.yaml
Deploy Prometheus and Grafana. Promtail depends on Prometheus given the ServiceMonitor resource. That is why the installation of Promtail must happen later on.
# Prometheus helm upgrade --install prometheus -n logging bitnami/kube-prometheus --version 6.6.0 --create-namespace # Grafana helm upgrade --install grafana grafana/grafana -n logging --version 6.13.5 --create-namespace -f - <<EOF datasources: datasources.yaml: apiVersion: 1 datasources: - name: Prometheus type: prometheus uuid: prometheus access: proxy url: http://prometheus-kube-prometheus-prometheus.logging.svc.cluster.local:9090 isDefault: true editable: true adminUser: admin adminPassword: password service: type: LoadBalancer sidecar: dashboards: enabled: true EOF
Finally, you need to install Promtail. Notice in the configuration the regex to catch the log traces you are targeting:
# Promtail helm upgrade --install promtail -n logging grafana/promtail --version 3.8.1 -f - <<EOF extraArgs: - -dry-run # This blocks the config: logLevel: info serverPort: 9080 # To expose the metrics lokiAddress: "http://this-will-be-ignored-due-to-dry-run" # To not deviate from the purpose of the article, you do not integrate with Loki snippets: pipelineStages: - match: selector: '{app="extauth"}' stages: - regex: # This is the regex expression: ".*(?P<passthrough_not_ready>error.*failed to create grpc client manager).*" - metrics: passthrough_not_ready: prefix: 'gloo_' type: Gauge # type: Counter description: "Total count of log traces when a passthrough service is not ready" source: passthrough_not_ready config: action: inc serviceMonitor: enabled: true interval: 10s scrapeTimeout: 5s EOF
Once you have Grafana ready, access it through your browser:
kubectl -n logging get svc grafana -o jsonpath='{.status.loadBalancer.ingress[0].*}'
You can use these credentials:
Username: admin Password: password
Create a dashboard with following specifications:
- Query:
count(gloo_passthrough_not_ready)
Panel configuration. The most relevant part is this:
You now have all the integration ready.
Set the dashboard to show the last 5 mins and you should see the panel:
Testing your configuration
Let’s verify that the dashboard shows an error once it occurs.
Apply the AuthConfig resource:
kubectl apply -f authconfig.yaml
Port-forward to access Promtail’s exposed metrics:
kubectl --namespace logging port-forward daemonset/promtail 9080
And check the metrics in your browser: http://127.0.0.1:9080/metrics
:You should be able to see your new metric
If you go to your new Grafana dashboard you should see the error:
Once you deploy the Authz service, there will be no more errors in the logs. After the event is propagated, the dashboard turns to No Error.
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/extauthz/ext-authz.yaml
The creation of the Alert to handle this Error is left to the reader.
Level up for more advanced metrics use cases
The case above is straight forward. A more complex scenario is to leverage Header Transformations and then to Enrich Access Logs. The goal here is to have that processed data in the logs. Once in there, you can create metrics out of them.
Examples of usage can be to perform Analytics to understand the usage of your system based on requests.
In an e-commerce system:
- How many times a day Product A (my-e-commerce.com/products/product-a) is more visited compared to Product B (my-e-commerce.com/products/product-b)
An analytics example with Istio and an eCommerce website will be published shortly in the official Grafana blog site. Stay tuned.
Thanks to Neeraj Poddar for his feedback!
BACK TO BLOGAbout the Author
