Netnod’s anycast DNS: 20 years of 100% availability
As stewards of one of the Internet's 13 DNS root-server clusters, Netnod employs a technology called "anycast" to make our service available from a large number of locations across the world. Netnod (or, to be precise, its then daughter company, Autonomica) was a pioneer of anycast technology, and this year marks the 20th anniversary of Netnod's first anycast deployment.
On 22 August 2003, at 17:01 in the afternoon, I sent the historic message above to my fellow system administrators at all twelve root-server operators, after having just fired up instance number two of I-root. 'Number one' had been in operation for twelve years by then, but going from one to two instances was the major step, as that suddenly gave the routers on the Internet more than one path to the target. This sent ripples through the Internet’s routing fabric as the new instance of I-root was added to routing tables, BGP route selection algorithm came into play, and routers determined the best route to use, and to forward to their peers.
We chose to install the second server with our good friends at the Finnish Internet Exchange Point FICIX in 'nearby' Helsinki, Finland. Helsinki and Stockholm are a mere 400 km apart, but there is a troublesome amount of water in between, commonly referred to as the Baltic Sea.
At this time, Virtual Machines were not a well-known and reliable concept, so we had to install an entire stack of physical servers performing the different, necessary tasks.
The systems were mounted in a rack at the exchange point and furnished with power and network connections to handle the expected DNS traffic, and to receive management instructions remotely from Stockholm.
A lot of planning went into the network configuration and into preparing the network setup at the existing instance in Stockholm to deal with dual-instance operation. We also prepared new server infrastructure in Stockholm to handle data distribution of the DNS root zone. The root zone is the database that root-servers serve data from. It is updated twice per day, and now we needed our own distribution facilities to furnish two instances as opposed to just one. Our plans were to increase that number from two to 20 over the course of two to three years.
The server in Helsinki was first started without access to the exchange point. It could only talk to the 'mothership' in Stockholm. Using that channel it was loaded with correct data. The last pieces (except for the very last one) of the networking puzzle were laid and double-checked.
Once we felt comfortable that everything was in place, we enabled the connection towards the exchange point, still without 'announcing' to the network that there was anything to reach behind our router. We established routing relations to some of the major service providers at the exchange point, still without announcing anything of interest.
After that, I started a small side program (tcpdump) to watch incoming DNS traffic from the exchange point. There was none, which was as it should be. With a certain thrill I then entered the command in the router in Helsinki to announce the existence and reachability of the local I-root instance to all the service providers, and tell them that they could send traffic 'here' instead of all the way to Stockholm.
As soon as I released my finger from the ENTER key, the tcpdump sprang to life, and DNS packets poured in from the Finnish service providers. A quick look at the old instance in Stockholm showed that everything still looked good. There was a small dip in the incoming traffic volume – traffic that now went to Helsinki instead. Success!
We still expected to see some oddities with traffic streams and routing, but there were precious few. We felt that this was to be our future.
Over the last 20 years, Netnod has deployed DNS root-server instances at well over 80 locations across the globe: from Helsinki in Finland, to Port Vila in Vanuatu, from Thimphu in Bhutan to Santiago in Chile, from Kigali in Rwanda to Ulaanbaatar in Mongolia. We operate the northernmost root-server of all, in Luleå, Sweden. Our main instance is still in Stockholm, Sweden, in a bombproof communications bunker. It was recently upgraded – again! – with stronger CPUs and 'fatter pipes' to the Internet, to remain our flagship instance.
To count the number of DNS queries we have responded to over the years is both impossible and pointless, but to give you a hint of the numbers: Netnod's current armada of root name servers responds to roughly 3,700,000,000 queries ... every single day! And that is without stirring up the dust.
The deployment of anycast reinforced the redundancy of our system. Since that day in August 2003, Netnod's root service has not been down, even once. Slightly degraded, yes, at certain points in time, but never down. That's 20 years of 100.0000 % availability. (Yes, that's right: there are no 9s in there!)
We are proud to provide this service for free as part of Netnod’s mission to work for the good of the Internet.
