Bender DNS Tutorial

July 23, 2018 · View on GitHub

This tutorial walks through the steps to create a simple DNS server with a load tester. If you already know how to Go's DNS libraries, skip to the "Load Testing" section to get started with Bender and DNS.

Getting Started

You will need to install and configure Go by following the instructions on the Getting Started page. Then follow the instructions on the How To Write Go Code page, particularly for setting up your workspace and the GOPATH environment variable, which we will use throughout this tutorial.

Next we need the DNS library for Go, which you can fetch using go get as:

cd $GOPATH
go get github.com/miekg/dns

Finally you will need the latest version of Bender, which you can get by running:

go get github.com/pinterest/bender

Writing DNS Server

This section will walk through the creation of DNS client and server, which we will use to test Bender in the following section. If you already have a DNS server you can use them instead.

In the following, all commands should be run from the $GOROOT directory, unless otherwise noted.

Creating the Go Package

Create a new Go package for your DNS server. We'll refer to this as $PKG in this document, and it can be any path you want. At Facebook, for example, we use github.com/facebook.

cd $GOPATH
mkdir -p src/$PKG/hellodns

DNS Server implementation

Now we will create a simple DNS server, that responds on A records with predefined addresses. First create a new directory:

mkdir -p src/$PKG/hellodns/server

Then create a file named main.go in that directory and add the following lines:

package main

import (
	"flag"
	"fmt"
	"log"

	"github.com/miekg/dns"
)

func handleRequest(w dns.ResponseWriter, r *dns.Msg) {
	m := new(dns.Msg)
	m.SetReply(r)
	for _, q := range m.Question {
		switch q.Qtype {
		// For any AAAA record answer with fd6c:1a5c:2b63::d8e9
		case dns.TypeAAAA:
			log.Printf("Query for \"%s\"", q.Name)
			rr, err := dns.NewRR(fmt.Sprintf("%s AAAA fd6c:1a5c:2b63::d8e9", q.Name))
			if err == nil {
				m.Answer = append(m.Answer, rr)
			}
		}
	}
	w.WriteMsg(m)
}

func main() {
	// Allow user to specify server port via flag
	var port int
	flag.IntVar(&port, "port", 5353, "server port")
	flag.Parse()

	// Attach handler function
	dns.HandleFunc(".", handleRequest)
	// Create and start the server
	server := &dns.Server{
		Addr: fmt.Sprintf(":%d", port),
		Net:  "udp",
	}
	err := server.ListenAndServe()
	if err != nil {
		log.Fatalf("Error: %s", err.Error())
	}
	defer server.Shutdown()
}

Run the server

Run these commands to compile and install the server:

go install $PKG/hellodns/server

In one terminal window, run the server as ./bin/server and in another terminal window run:

dig -p 5353 AAAA @localhost test.

The server should log receiving the query for "test." and client should print the answer similar to this one:

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -p 5353 AAAA @localhost test.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48142
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;test.                          IN      AAAA

;; ANSWER SECTION:
test.                   3600    IN      AAAA    fd6c:1a5c:2b63::d8e9

;; Query time: 0 msec
;; SERVER: ::1#5353(::1)
;; WHEN: Tue Jul 17 02:44:41 PDT 2018
;; MSG SIZE  rcvd: 54

Load Testing

Now that we have an DNS server we can use Bender to build a simple load tester for it. This section uses the same directories and packages as the previous section, but it is easy to use the same instructions for any DNS server. The next few sections walk through the various parts of the load tester. If you are in a hurry skip to the section "Final Load Tester Program" and just follow the instructions from there.

Intervals

The first thing we need is a function to generate intervals (in nanoseconds) between executing requests. The Bender library comes with some predefined intervals, including a uniform distribution (always wait the same amount of time between each request) and an exponential distribution. In this case we will use the exponential distribution, which means our server will experience load as generated by a Poisson process, which is fairly typical of server workloads on the Internet (with the usual caveats that every service is a special snowflake, etc, etc). We get the interval function with this code:

intervals := bender.ExponentialIntervalGenerator(qps)

Where qps is our desired throughput measured in queries per second. It is also the reciprocal of the mean value of the exponential distribution used to generate the request arrival times (see the wikipedia article above). In practice this means you will see an average QPS that fluctuates around the target QPS (with less fluctuation as you increase the time interval over which you are averaging).

Request Generator

The second thing we need is a channel of requests to send to the DNS server. When an interval has been generated and Bender is ready to send the request, it pulls the next request from this channel and spawns a goroutine to send the request to the server. This function creates a simple synthetic request generator:

func SyntheticDNSRequests(n int) chan interface{} {
	c := make(chan interface{}, 100)
	go func() {
		for i := 0; i < n; i++ {
			msg := new(dns.Msg)
			msg.SetQuestion(dns.Fqdn(fmt.Sprintf("test%d", i)), dns.TypeAAAA)
			c <- msg
		}
		close(c)
	}()
	return c
}

Request Executor

The next thing we need is a request executor, which takes the requests generated above and sends them to the service. We will use a helper function from Bender's dns library to do most of the work (connection management, error handling, etc), so all we have to do is write code to verify the request:

var responseIP = net.ParseIP("fd6c:1a5c:2b63::d8e9")

func validator(_, resp *dns.Msg) error {
	if len(resp.Answer) != 1 {
		return fmt.Errorf("response missing answer for query")
	}
	answer, ok := resp.Answer[0].(*dns.AAAA)
	if !ok {
		return fmt.Errorf("invalid answer type AAAA expected")
	}
	if !responseIP.Equal(answer.AAAA) {
		return fmt.Errorf("invalid address: %s, expected %s", answer.AAAA, responseIP)
	}
	return nil
}

exec := http.CreateExecutor(nil, validator, "localhost:5353")

This validates that the response has an answer with predefined address.

Recorder

The last thing we need is a channel that will output events as the load tester runs. This will let us listen to the load testers progress and record stats. We want this channel to be buffered so that we can run somewhat independently of the load test without slowing it down:

recorder := make(chan interface{}, 128)

The LoadTestThroughput function returns a channel on which it will send events for things like the start of the load test, how long it waits between requests, how much overage it is currently experiencing, and when requests start and end, how long they took and whether or not they had errors. That raw event stream makes it possible to analyze the results of a load test. Bender has a couple simple "recorders" that provide basic functionality for result analysis and all of which use the Record function:

NewLoggingRecorder creates a recorder that takes a log.Logger and outputs each event to it in a well-defined format.
NewHistogramRecorder creates a recorder that manages a histogram of latencies from requests and error counts.

You can combine recorders using the Record function, so you can both log events and manage a histogram using code like this:

l := log.New(os.Stdout, "", log.LstdFlags)
h := hist.NewHistogram(60000, 10000000)
bender.Record(recorder, bender.NewLoggingRecorder(l), bender.NewHistogramRecorder(h))

The histogram takes two arguments: the number of buckets and a scaling factor for times. In this case we are going to record times in milliseconds and allow 60,000 buckets for times up to one minute. The scaling factor is 1,000,000 which converts from nanoseconds (the timer values) to milliseconds.

It is relatively easy to build recorders, or to just process the events from the channel yourself, see the Bender documentation for more details on what events can be sent, and what data they contain.

Final Load Tester Program

Create a directory for the load tester:

mkdir -p src/$PKG/hellobender

Then create a file named main.go in that directory and add these lines to it:

package main

import (
	"fmt"
	"log"
	"net"
	"os"
	"time"

	bdns "facebender/protocols/dns"

	"github.com/miekg/dns"
	"github.com/pinterest/bender"
	"github.com/pinterest/bender/hist"
)

var responseIP = net.ParseIP("fd6c:1a5c:2b63::d8e9")

// SyntheticDNSRequests generates 100 dummy requests to the dns server
func SyntheticDNSRequests(n int) chan interface{} {
	c := make(chan interface{}, 100)
	go func() {
		for i := 0; i < n; i++ {
			msg := new(dns.Msg)
			msg.SetQuestion(dns.Fqdn(fmt.Sprintf("test%d", i)), dns.TypeAAAA)
			c <- msg
		}
		close(c)
	}()
	return c
}

func validator(_, resp *dns.Msg) error {
	if len(resp.Answer) != 1 {
		return fmt.Errorf("response missing answer for query")
	}
	answer, ok := resp.Answer[0].(*dns.AAAA)
	if !ok {
		return fmt.Errorf("invalid answer type AAAA expected")
	}
	if !responseIP.Equal(answer.AAAA) {
		return fmt.Errorf("invalid address: %s, expected %s", answer.AAAA, responseIP)
	}
	return nil
}

func main() {
	intervals := bender.ExponentialIntervalGenerator(10)
	requests := SyntheticDNSRequests(100)
	exec := bdns.CreateExecutor(nil, validator, "localhost:5353")
	recorder := make(chan interface{}, 100)

	bender.LoadTestThroughput(intervals, requests, exec, recorder)

	l := log.New(os.Stdout, "", log.LstdFlags)
	h := hist.NewHistogram(60000, int(time.Millisecond))
	bender.Record(recorder, bender.NewLoggingRecorder(l), bender.NewHistogramRecorder(h))
	fmt.Println(h)
}

Run Server and Load Tester

Run these commands to compile and install the server and load tester:

go install $PKG/hellodns/server
go install $PKG/hellobender

In one terminal window, run the server as ./bin/server and in another window run the load tester as ./bin/hellobender. You should see a long sequence of outputs in the server window and a final print out of the histogram data in the other window.