Devhost Part 02
The Continuation…⌗
In a previous episode we looked at a way to solve the problem of running multiple services that might need to talk to each other in a sane way. The solution was to run each service in a Docker container and to put any container from the service that you want exposed on a common network that is accessible via a proxy service. We also came up with a list of three possible proxy services to use and tested them out (as well as we could at the time anyway) and decided on two options to continue experimenting with. We also came up with two questions that we needed to answer to come up with something that is actually usable.
To quickly recap, the proxy services we want to try further are Traefik and Envoy and the two questions we need to answer are:
- Can we run a simple DNS service in a container to provide unique domain names for each service? This would make running Traefik with host based rules more manageable.
- Is there a basic control plane that we can use to coordinate service discovery for Envoy? Istio works but is incredibly complicated to set up.
Since I don’t want blog posts to get too long, I’m only going to focus on the DNS aspect of things for now. So with that in mind, it’s time for some research!
Research⌗
The goal of having a DNS service included in the development environment is to provide a domain that you have authoritative control over to easily assign a domain name to each service you might run. So if you had two services named ServiceA and ServiceB, you would have servicea.<my-local-domain> and serviceb.<my-local-domain>. So there are two other ways you could handle this without going to the work of setting up a DNS service.
The first way you could do this is to just use the ubiquitous /etc/hosts file. You would then just add the service and whatever domain name you wanted to the file and instantly get resolution to your service. Sounds great and is incredibly simple but now you have to maintain that file. Every time you add or remove a service you need to make a change to the file. I’m a developer and therefore lazy. I don’t want to have to manage something if I don’t have to so I’ll come up with a more complicated solution if it means less work for me later.
The second possible solution is to use the *.localhost TLD. By default this is provided for me by my Linux system (#linuxmasterrace) but I have a very strong suspicion that I can’t safely rely on this being present on other systems. I’m all about making sure things work on as many systems as possible if only because I’ve been in a situation where I’ve had to figure out how to make something work on Mac or Windows after knowing that it works on a Linux environment. If I plan for cross-platform support from the beginning I don’t have to worry about that happening. The other problem with using the localhost TLD is that it may not be usable when doing things like playing with OAuth. I’ve had at least one OAuth provider that required a public gTLD before it would allow me to do authentication.
An additional reason to go with a managed DNS service is that I can do some additional tricks such as hijacking a domain if I want/need to (think of all the spam/ads you could block with this!). A realistic usage that you might encounter is if you have to reverse engineer a connection to a service you don’t control or if you’re using a service that is hard coded to hit a specific URL for a service you can run but the URL is configured for a different environment. Depending on the DNS software used, you could also easily ship these tricks to other people running a similar environment without having to provide instructions on how to patch their host file or other random system files. Just give them some configuration files and they’re off and running.
So we’ve decided on going the DNS route, what are our options? The three implementations that come to mind for me are dnsmasq, BIND, and CoreDNS. All three require an additional cap because of the way the resolver wants to talk to them so we need to keep that in mind. The first two are pretty traditional DNS offerings that can be easily containerized and the third was developed with containers in mind.
Note for Linux Users⌗
My system needed to have the /etc/nsswitch.conf file updated to actually pass DNS requests to my internal DNS service. The change was pretty minor and the resulting config looked something like this:
passwd: compat
group: compat
shadow: compat
gshadow: files
# This is where the relevant change happened.
hosts: files dns mdns4_minimal [NOTFOUND=return] myhostname
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
Basically I had to just move the dns option in front of the mdns4_minimal bit.
dnsmasq⌗
dnsmasq is a common player in the Linux ecosystem. It combines DNS, Resolver, and DHCP components all in one package which makes it great for adding networking to a Linux computer. Because of how common it is, there are no shortage of tutorials and guides out there on top of the man pages explaining how to work with dnsmasq.
I found what seemed like a reasonable implementation of a dnsmasq Docker image provided by andyshinn and used that to generate a default configuration to build off of. I then updated my docker-compose.yml file to include this image.
version: "3.5"
services:
# ... proxy configuration ...
dns:
image: andyshinn/dnsmasq:2.78
restart: always
# This is needed to be able to bind a socket in a way that DNS resolvers
# expect.
# http://man7.org/linux/man-pages/man7/capabilities.7.html
cap_add:
- NET_ADMIN
volumes:
- type: bind
source: ./dns/dnsmasq.conf
target: /etc/dnsmasq.conf
read_only: true
- type: bind
source: ./dns/dnsmasq.d
target: /etc/dnsmasq.d
read_only: true
ports:
# This format is required to bind a port with two protocols.
- "127.0.0.1:53:53/udp"
- "127.0.0.1:53:53/tcp"
You’ll notice that on top of mounting the config file that I generated into the container, I’m also mounting a directory to /etc/dnsmasq.d which I added to allow for an easier time creating special configurations when needed.
I then spent a bunch of time updating that dnsmasq.conf file that I generated. After a few tweaks I had it forwarding anything in the *.dev.local space (that was the domain that I picked for now) to my Traefik container that I’ve been experimenting with. This worked great but I was bothered by all the other stuff that was running in dnsmasq and the config file is very long and has a lot in it. It seemed wrong to have a big piece of software that has most of its functionality disabled. It was because of how much other stuff dnsmasq does that I decided it wasn’t a good fit.
BIND⌗
Next up on the chopping block is BIND. BIND intrigued me as an option because it’s become the defacto standard for DNS implementations and has been around forever (since DNS was invented!) so you know it’s got a lot of support. There’s also a ton of information on setting it up. The only downside I’ve found with it is that there’s no decent Docker images for it yet but thankfully putting one together wasn’t terribly hard.
FROM alpine:3.7
RUN set -ex \
&& apk update \
&& apk add --no-cache \
bind \
&& rm /var/cache/apk/*
EXPOSE 53/udp 53/tcp
ENTRYPOINT [ "named", "-g" ]
So with the Docker image implemented and out of the way I integrated it with my docker-compose.yml file.
version: "3.5"
services:
# ... proxy configuration ...
dns:
build:
context: ./dns
restart: always
# This is needed to be able to bind a socket in a way that DNS resolvers
# expect.
# http://man7.org/linux/man-pages/man7/capabilities.7.html
cap_add:
- NET_ADMIN
volumes:
- type: bind
source: ./dns/named.conf
target: /etc/bind/named.conf
read_only: true
- type: bind
source: ./dns/zones
target: /etc/bind/zones
read_only: true
ports:
# This format is required to bind a port with two protocols.
- "127.0.0.1:53:53/udp"
- "127.0.0.1:53:53/tcp"
Apart from mounting the config file (named.conf) into the container, I’m also mounting a directory called zones that contains all the domain/zone configuration that we need. Below you’ll see the named.conf file contents. It’s pretty simplistic and just defines the loopback address, the actual domain we want to control, and a reference to the external root name servers.
options {
directory "/etc/bind/zones";
};
zone "dev.local" in {
type master;
file "db.dev.local";
};
zone "0.0.127.in-addr.arpa" in {
type master;
file "db.127.0.0";
};
zone "." in {
type hint;
file "db.cache";
};
Then we have the db.dev.local file which contains the authoritative zone information for dev.local.
$TTL 3h
dev.local. IN SOA dns. root.dns. (
1 ; Serial
3h ; Refresh rate
1h ; Retry rate
1w ; Expiry
1h ; Negative caching TTL
)
; Name servers
dev.local. IN NS dns.
; Records
*.dev.local. IN A 127.0.0.1
And finally the loopback pointer config file.
$TTL 3h
0.0.127.in-addr.arpa. IN SOA dns. root.dns. (
1 ; Serial
3h ; Refresh time
1h ; Retry delay
1w ; Expiry
1h ; Negative TTL
)
0.0.127.in-addr.arpa. IN NS dns.
1.0.0.127.in-addr.arpa. IN PTR localhost.
There is a third zone file that I reference in named.conf and that’s db.cache which contains information about the root nameservers to use for all requests that we don’t know how to handle ourselves. I do have this file but it’s just a copy of what can be found at the official source. It didn’t seem necessary to copy/paste that file here.
Now this setup works but it has a small downside which is that all DNS lookups on this computer are processed by this container. Including lookups for domains that are external (like google.com). This isn’t ideal since having this container go down will cause issues with normal requests (mostly having them timeout the first time) and I could fix this by having BIND just respond by saying it doesn’t know how to find the requested domain if it isn’t *.dev.local.
Overall this isn’t a bad option. It does require a few more files than I might like but BIND is a solid choice.
CoreDNS⌗
CoreDNS is a newer DNS implementation (relative to the other options explored) and was designed specifically for the new Cloud based world we live in nowadays (one might say its web scale…). One of the really nice things about CoreDNS is that it’s sort of plugin based which makes it more customizable. I say sort of because any plugin you want to use needs to be compiled into the binary (this will likely change once plugins in Golang take off more). You can compile in external plugins though so it is still flexible.
Unlike dnsmasq and BIND, CoreDNS provides a Docker image for us so we don’t need to do any work to set that up and can just use it in our docker-compose.yml file directly.
version: "3.5"
services:
# ... proxy configuration ...
dns:
image: coredns/coredns:1.0.6
restart: always
command: -conf /etc/coredns/Corefile
# This is needed to be able to bind a socket in a way that DNS resolvers
# expect.
# http://man7.org/linux/man-pages/man7/capabilities.7.html
cap_add:
- NET_ADMIN
volumes:
- type: bind
source: ./coredns/Corefile
target: /etc/coredns/Corefile
read_only: true
- type: bind
source: ./coredns/zones
target: /etc/coredns/zones
read_only: true
ports:
# This format is required to bind a port with two protocols.
- "127.0.0.1:53:53/udp"
- "127.0.0.1:53:53/tcp"
As before, I’m mounting the configuration file as well as a directory for any zones that we might want or need. Right now the config file is hardcoded to load a single zone but if I want to pursue this option further, I’m fairly certain I could make it smart enough to dynamically load zones. Below is the Corefile that serves as the primary config file for CoreDNS.
dev.local {
log
errors
file /etc/coredns/zones/db.dev.local
}
All it does is tell CoreDNS to log requests and errors for any domain in the dev.local zone. It pulls the authoritative information from the db.dev.local file which looks like this.
$ORIGIN dev.local.
@ 30 IN SOA dns. root.dns. (
1 ; Serial
3h ; Refresh rate
1h ; Retry rate
1w ; Expiry
1h ; Negative caching TTL
)
30 IN NS dns.
; Records
* IN A 127.0.0.1
It’s pretty much the same as the zone file I put together for BIND which was nice since it meant I didn’t have to do much work to get this functioning. One key difference to how I set CoreDNS up compared to BIND is that CoreDNS only knows how to respond to requests in the dev.local domain and will return an error for anything else. Having CoreDNS fail early for external requests doesn’t seem to have any negative impact on requests that I’ve noticed.
Results⌗
So where does this leave us? We have three viable options that all work. I’m going to eliminate dnsmasq as an option just because it seems like overkill and provides way more than I actually need. That just leaves BIND and CoreDNS. Both of them provide the same functionality and both of them work well.
I guess the big differences between them lie in their usability. BIND has been around since the beginning of DNS so there’s a lot documentation and experience to draw upon for running it. Then again, I’m not running a production system that needs perfect results. CoreDNS on the otherhand is the hot new thing on the block and while it doesn’t have a ton of documentation just yet it is under very active development and is likely the direction that the industry is headed. It was designed to be flexible and provides quite a bit more choices for deployment than BIND does. Further, the codebase for CoreDNS is much more approachable which means that I can jump into it or make changes if I need.
Given all that, I’m going to continue forward with CoreDNS as the DNS provider for my setup!
Next Time⌗
In the next episode we’ll look at setting up a system for generating TLS certificates in as simple a manner as possible.