🖥 10 New Things I Discovered when Building a Virtual DNS Server

Ah, the traditional bank holiday weekend project. Fix the shed, paint the bedroom or build a virtual DNS server.

I was inspired by Brian Christner’s post “How a Single Raspberry Pi made my Home Network Faster”. In his case, he used a physical Raspberry Pi and Docker. I considered the same approach, but in the end since I have a Linux server (an old laptop I inherited from work) already running in my flat (it mainly runs Docker containers for various things), why not use that?

It’s unfortunately not possible to just implement the Docker-based approach from Brian’s article, as I’m using the host machine for other activities, and DNS needs exclusive use of specific ports. In essence, it has to be a machine dedicated to the task.

The approach I went with was to create a Virtual Machine which sits on the network with its own IP addresses. To any other machine on the network, it shows up as a dedicated DNS server.

In aiming to automate this as much as possible, I utilised Vagrant with a sprinkling of Ansible. You can try out the main project on GitHub and so benefit from speedier DNS using the same approach I took.

But as with all personal projects, it’s the interesting things you learn along the way which can be the most exciting part, and I’d be remiss not to share my findings about these tools with the class, so let’s dive in!

1. The “V” in Vagrant is for “adVenturing”

If you take one thing away from this post, let it be this: Vagrant allows you to spin up a fresh VM for experimentation in just a few seconds, with almost zero effort.

Back in the olden days if you were a Windows user who wanted to play around with Linux, you had to partition your hard drive (which was in general a dangerous thing to do) and install Linux on a separate partition. You’d then have to set up dual booting. This was a weekend project in itself, or maybe even a holiday project if it all went pear-shaped.

When PCs became powerful enough to run Virtual Machines, you could install Linux within Windows, safely. Of course, this involved downloading a Linux image, manually creating the VM, installing Linux, etc. Then, you have to know its IP in order to SSH into it, and after that you were good to go. But, it was still a manual process.

But move into the current day, and we have Vagrant. With a few lines of code, you can create a fresh Virtual machine and start it with a single command: vagrant up. It really is that straight forward!

Want to give it a try? No need to know any Ansible or devops mysticism. Just create a file named Vagrantfile like in an empty directory like so:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"
end

Then run vagrant up and vagrant ssh to enter the VM. Shut it down with vagrant halt and delete it with vagrant destroy. The VM is persistent, so you can play around with it to your heart’s content.

(prerequisites are VirtualBox and Vagrant)

2. Vagrant Wouldn’t Install

First issue I ran into was in trying to install Vagrant on Ubuntu (18.04.2 LTS) where I was greeted with the following error:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 vagrant : Depends: bsdtar but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

A bit of Googling around revealed the workaround is to download and install Vagrant manually, which is pretty straight forward. If you run into the same error I did, that’s your best option.

3. Which Came First: The Python or the Egg?

One of the great things about Ansible over similar technologies such as Puppet is it doesn’t require a server-side component. In fact, all you need on the server is a running SSH daemon, Python installed, and that’s it.

But, I found myself in a bit of a chicken and egg situation as I was building this thing. When you pull down a basic image of Ubuntu Server via Vagrant, it doesn’t have Python installed by default. Typically, one of the first things you want your Ansible playbook to do is install dependencies on a fresh VM. But how do you install Python using Ansible when you can’t run Ansible without Python?

Lucky for me, I came across this post on Elf Sternberg’s blog, where he discusses Ansible’s “raw” module. This lets you run commands directly, without any of the sugar Ansible usually supplies.

This is one of the only scenarios I can see where you’d want to run raw commands, as normally you’d run the apt module to install things (or apk if you’re using Alpine Linux) but it’s enough to get Python installed so the rest of your playbook can run:

---
- name: Install Python
  hosts: pihole
  become: True
  gather_facts: False
  tasks:
    - name: install python
      raw: apt-get install python -y

This is functionally the same as doing it the “correct” way:

---
- name: Install Python
  hosts: pihole
  become: True
  gather_facts: True
  tasks:
    - name: install python
      apt:
        name: python

Another approach would be to do this directly in your Vagrantfile immediately after provisioning the VM, leaving Ansible out of it completely. This isn’t an approach I explored, but I imagine it would work just as well. The Ansible raw technique of course has the advantage of working outside of Vagrant.

4. Vim Highlighting in a Vagrantfile

Take a look at a lot of Vagrantfiles on Github, and you’ll see something like this at the top of them:

# -*- mode: ruby -*-
# vi: set ft=ruby :

These magic lines are a directive to Vim and Emacs to treat the Vagrantfile as if its filename was postfixed with .rb, as after all Vagrant is written in the language of Ruby. However, I tend to avoid doing this as these comments don’t really serve much purpose to people using other text editors.

A better approach is to configure your own Vim setup to recognise Vagrantfiles as containing Ruby, and highlight the syntax accordingly. You can do this by adding the following lines to your .vimrc:

" File type Vagrantfile (uses ruby syntax highlighting)
augroup vagrant
  au!
  au BufRead,BufNewFile Vagrantfile set filetype=ruby
augroup END

(Source: Gist)

5. Using Vagrant Plugins

When I started this project on a bank holiday Sunday afternoon, I originally wanted to use Alpine Linux as the distribution, because it’s light-weight and elegant. I quickly realised Pi-hole would only function on a Debian based system, and made the switch to Ubuntu. But before that I learnt that Vagrant doesn’t honour all distributions with equal nicety.

Consider this simple Vagrantfile to provision Alpine:

Vagrant.configure("2") do |config|
  config.vm.box = "maier/alpine-3.7-x86_64"
end

Run vagrant up and everything goes like a charm. But try and bring down the VM, and well… it doesn’t work and your VM stays up:

$ vagrant halt
==> testy: Attempting graceful shutdown of VM...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

shutdown -h now

Stdout from the command:

Stderr from the command:

bash: line 4: shutdown: command not found

So, what’s going on here? When running vagrant halt, Vagrant attempts to stop the machine using the bog-standard shutdown command, but it doesn’t exist, and so it falls over and you’re now forced to kill the VM with vagrant halt --force, which as the help page suggests is equivalent to pulling the power cable. Not what you want.

Certainly, you could try and install the shutdown command, but Vagrant includes a plugin architecture, and good kind people have written plugins to help you handle non-standard distributions. In the case of Alpine, you can run:

vagrant plugin install vagrant-alpine


And now you can safely shutdown your beloved VM with a simple vagrant halt.

If you want to see which other Vagrant plugins are available, have a look at this repo for the full spectrum.

6. Not All Images are Created Equal

That bank holiday Sunday I mentioned quickly turned into a bank holiday Monday as no matter what I did, I couldn’t get my VM to allow me to SSH into it.

I’d set the following to ensure my VM appeared on the network with its own IP(s):

config.vm.network "public_network"


For the life of me I couldn’t figure out why it wouldn’t work, as in private_networkmode everything was fine.

As I spent hours trying to debug the Vagrantfile and VM. I began to lose patience, and as a last resort I tried using a different image of the same distribution. The problem just went away.

The lesson here is to try another image if something as basic as SSH just isn’t working.

7. Speeding Up Playbooks by Skipping Facts

In my travels around Google searching for answers to other problems, I came across Abhijeet Kamble’ great blog post 10 Things you should start using in your Ansible Playbook. Some of these tips I knew, most I didn’t. The one which stood out was the item on when to use facts gathering, and when not to.

In the case of my own playbooks for this project, facts gathering just wasn’t needed. And removing this step sped up the playbook considerably, as well as reducing the on-screen output.

8. Digging for Routes

Once you’re ready to test your Pi-hole VM, you’ll want to change your router or laptop’s DNS server to point at your VM, and here you’ll hit a dilemma.

Initially, you should set up a secondary DNS server (eg: 1.1.1.1) to fall back on when your VM fails or can’t be reached. For example when you reboot the server and what-not. But are your DNS requests hitting Pi-hole, or are they hitting the backup DNS server?

Enter the dig command, which may be run against any URL:

dig arstechnica.com


It’ll tell you which DNS server you’re hitting. For my local network, here’s what I see when I successfully hit Pi-hole as my primary DNS at 10.0.1.35 on my local network:

;; Query time: 101 msec
;; SERVER: 10.0.1.35#53(10.0.1.35)
;; WHEN: Thu May 23 22:25:41 EDT 2019
;; MSG SIZE rcvd: 74

And if I halt the VM, I fall back on the secondary DNS:

;; Query time: 15 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu May 23 22:28:10 EDT 2019
;; MSG SIZE  rcvd: 60

Use this technique to help debug your VM and know whether you’re hitting Pi-hole or not.

9. Restart on Restart

If you work with Docker containers a lot, you might know they can be configured to always start when the host machine is rebooted. Wouldn’t it be nice if you could do this with your VM as well? It turns out, you can.

Just create a file on the host machine at sudo vim /etc/rc.local:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
/bin/su -c 'cd /path/to/pihole-vagrant && /path/to/vagrant up' YOUR_USERNAME
exit 0

If you’re not sure where vagrant lives, run:

which vagrant


And finally give it execute permissions:

sudo chmod +x /etc/rc.local


One more thing you might need to do, is to remove the manual prompt which asks you to select the interface when running vagrant up. For example, in your Vagrantfile change:

config.vm.network "public_network", auto_config: true


to:

config.vm.network "public_network", bridge: 'enp0s25', auto_config: true


And replace enp0s25 with the name of your chosen interface.

10. Moo!

This is one I knew about long before this project, but it’s great fun so I’ll add it as the final item. Consider the following Ansible Playbook:

---
- name: This is a hello-world example
tasks:

    - name: Create a file called '/tmp/testfile.txt' with the content 'hello world'.
    copy:
        content: hello world
        dest: /tmp/testfile.txt

Which outputs this when you run it:

TASK [Create a file called '/tmp/testfile.txt' with the content 'hello world'.] ***
changed: [default]

DULL! Try installing cowsay on the host machine with sudo apt-get install cowsay and then run the playbook again:

_________________________________________________________
/ TASK [Create a file called '/tmp/testfile.txt' with the \
\ content 'hello world'.]                                 /
 ---------------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

changed: [default]

Much more interesting! But, if you get bored of seeing the same cow over and over, try running export ANSIBLE_COW_SELECTION=random before starting your playbook. I’ll leave it to you to try it for yourself and see what happens!

(When you get bored of the cows, switch them off with export ANSIBLE_NOCOWS=1, and read more about this very important feature of Ansible here)

Feel free to share your own tips in the comments, or let me know how I can improve on my own approach!