In this article, we are going to deploy name servers and secure their DNS traffic internally and externally. In doing so, we will secure our traffic for the sake of privacy but also security. Making these changes will have an impact on eliminating things like surveillance, tracking, and even spoofing. Securing your DNS traffic and services is of critical importance.
If left unmitigated, your DNS traffic and services could be used as a tunnel to pass data outside of your network as seen in DNS Tunneling. A lack of restricted access to resolvers could allow poisoning in DNS pools or DNS Spoofing. A lack of control of DNS requests would provide an avenue of DDOS to your services via DNS Amplification. To these points, we will leverage Istio to control and reduce the attack surface in our DNS environment.
At Solo.io, we see a lot of interesting scenarios around security use cases and have solved a lot of customer and community issues. In this blog, the intent is to use tools that are all readily available in the open source community. We also see a lot of communication as it pertains to developers and securing their applications, but rarely using a service mesh to solve infrastructure security and availability.
For this project we will be using Pi-hole, CloudflareD, netshoot, and Istio. We’re using these pieces because:
- Pi-hole is a great tool to administrate and control DNS requests including blocking ads
- Cloudflare daemon is used to connect to any available authoritative DNS provider capable of https
- Netshoot is simply used as a tool to verify those requests as it includes dig and nslookup by default.
The parts and pieces
We are going to be using kubernetes, Istio, Pi-hole, and CloudflareD in order to set up a path that allows an entirely encrypted path from DNS request origination to the answer received from the authoritative name server. We will also be deploying everything using a microservice architecture and securing our internal pod traffic using mTLS.
While we could have eliminated some of that extra egress TCP overhead by making a recursive or authoritative name server, for now it’s out of scope of this example as it involves setting up something like unbound adding irrelevant complexity. See below for how the flow would look in Kiali.
As seen in the graph, Pi-hole has two services. One of those services is the administrative web-gui that is exposed to the public internet via the Istio ingress gateway and the TCP DNS service only available in the cluster. In this scenario, our DNS server will not interfere with existing Istio DNS proxying and will serve as a supplemental DNS server.
Since we are restricting our external DNS communication through this flow to TCP, considerations will need to be taken to enable pods to use TCP. Setting TCP for DNS can be done in several ways including forward with force_tcp in CoreDNS or amending resolv.conf with the nameserver and use-vc option. Pi-hole will call the CloudflareD pod via its service and CloudflareD will make requests over https to Cloudflare’s name servers.
How to make it happen
Before applying any of the manifests, make sure you have the istioctl binary installed. We will be using it along with its installation mechanism. After the binary is in place, install Istio in your cluster using istioctl install --set profile=default.
Secure
Next, we will install the manifests located in this repository. Inside it are the namespaces, services, deployments, config maps, and Istio manifests we need. Take note of the labels on these objects. They are needed in order to tell Istio what to apply to and where. There is also a manifest that sets mTLS to “strict,” forcing the correctly labeled objects to connect securely.
Control
Now we should have all of the pieces up and running. Optionally, we can define an authorization policy to the gateway that will restrict access to the pihole application admin interface from the public internet to just one IP. In the authorization policy example shown in the previously mentioned repository, we use ipBlocks as it correlates to a GKE load balancer being a network type. Also, to be certain my source IP is preserved, we patch the istio-ingressgateway service using the following command kubectl patch svc istio-ingressgateway -n istio-system -p {"spec":{"externalTrafficPolicy":"Local"}}' .
In doing so, the best practice is to deploy the istio-ingressgateway pod as a daemonset to be sure the request is served from any host that received traffic, however, that is out of scope for this demonstration.
Observe
Everything should now be up and running and ready to observe! Keeping with the theme of simplicity, we are going to deploy a default set of applications from the Istio repository.
Kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/grafana.yaml
Kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/jaeger.yaml
Kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/kiali.yaml
Kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/prometheus.yaml
The above commands should have installed a demonstration version of Grafana, Jaeger, Kiali, and Prometheus. We will install all these tools in order to explore tracing, graphs, and more, but for the sake of this article, we will look at Kiali namely.
In order to produce the graph seen earlier in the demonstration, let’s open Kiali using istioctl dashboard kiali. Doing so will start a port forward and open a browser window directly into the interface. From there, navigate to “Graph” on the left panel and select the “pihole” namespace. For extra visualization, click the display drop down and check the boxes for security and traffic animation.
To get traffic generated into the DNS services, I chose to use netshoot as it has everything we need for this purpose. Running the following will start a semi-permanent pod in your cluster to connect to in order to run DNS queries.
Once the pod is up and you are connected, running the following commands can generate the DNS queries we need.
Both of these commands accomplish the same task, which is to specifically query Pi-hole using TCP. Now that you have made some requests, check back on the Kiali dashboard for your graph to be populated.
Conclusions and next steps
At this point, you should have something stood up that you can experiment with in regards to Istio, but can also provide you an ad blocking service that you may not have had before. Even better – all those DNS requests are being sent securely end-to-end, reducing your attack surface.
Interested in this solution and want to learn more? Reach out to any of us at Solo.io on our public Slack channel and we would be happy to suggest further learning paths. We have guided courses and frequent public webinars that can help you find success in using service mesh, eBPF, API Gateways, and more!
I also recommended downloading our new white paper: Achieve Compliance, Zero Trust with Istio Ambient Mesh.
