It was a year ago today that the internet stumbled and nearly fell, when a human error caused large swathes of the InterNic database – effectively the address book of the internet – to be overwritten with blank entries. It meant that domain names such as ibm.com could not be mapped to the website’s IP address and therefore could not be found and email addresses ending ion .com and .net could not be accessed for a large part of the day. But a year on a lot of things have changed at Network Solutions Inc, the company that looks after the InterNic under a contract from the US government that is in the process of winding down now, and will end September 30. There are 13 root servers around the world (‘A’ through ‘M’) that each contain an identical copy of the root zone file for each top-level domain. NSI maintains the A root server, which contains the master root – it’s an unimposing dual-processor Sun Microsystems Inc Ultra Enterprise 2000. The root zone files are propagated out from the A server each day. Of the other 12 servers, nine are in the US with one each in Sweden, England and Japan. In addition to A, NSI also looks after M. Last year’s snafu happened when a customer service representative at NSI entered unusual data into a field that the system had not seen before. The data was inserted into a record in the Computer Associates International Inc Ingres database. At night when the data was pulled from the database to form the root zone file, it hit the bad data and truncated the record at that point. The zone generation software did not suspect that such a short return file from Ingres was abnormal, and loaded it into the root servers as usual. A system administrator noticed the problem from home, called an operator and alerted them to it, but the truncated root zone file went out anyway. That gradually corrupted the other servers that rely on A for the correct information. It did not affect all 13, but some of those that were affected were not righted for up to two days, which is more to do with the regularity of their maintenance than anything else. It should be remembered however, that most of the servers are maintained by volunteers. The person responsible was not fired, but quit soon after. On a recent tour of NSI’s facilities in a secret location about a mile from its Herndon, Virginia headquarters, the company’s senior VP engineering Dave Holtzman explained some of the measures the company has taken since last year to avoid a re- occurrence. NSI is about half way through the process of switching from Ingres to Oracle, mainly because Oracle supports other software that NSI uses better, such as network management. And, according to Holtzman, its maintenance releases are more robust. Along with such things as better training for everyone involved, the company has taken more specific measures such as ensuring that a senior person is on duty every night and that there are always two people involved in the updating process – one to do it and one as quality control. It has also introduced a staging area for the zone files to detect any problems before they are released and has developed a program to run against the files that detects anything unusual. At present the company has about 80 people in its engineering team, and that will rise to 105 before too long, says Holtzman. NSI feels sure its facilities are up to shouldering such a responsibility as maintaining the internet’s root zone system. It even has a diesel tank outside with enough furl to run the whole facility for at least four days and a room full of batteries that kick in straight away in the event of a power failure, running everything for about seven seconds, leaving enough time for the diesel power to kick in. Within a couple of years, said NSI senior VP Don Telage on the recent tour, all such facilities are likely to be placed in underground bunkers, such is their strategic importance. NSI’s stock is traded on Nasdaq and it is 80%-owned by Science Applications International Corp (SAIC), which is also publicly traded.
