There's nothing like a slowdown in the earth's rotation to ruin a Saturday night.
System administrators around the country were called away from dinner dates and gaming sessions on Saturday after an extra second was added to the evening by the world's timekeepers. That itty bitty adjustment triggered the widespread failure of computers that keep the world's websites online. Among the sites to have problems: Gawker, LinkedIn, Mozilla, Reddit, StumbleUpon and Yelp, plus countless others.
The Clock Tower of Parliament which houses Big Ben (Image credit: Getty Images Europe via @daylife)
The addition of the so-called "leap second," wasn't a surprise. It was scheduled months in advance by the timekeepers at the International Earth Rotation and Reference Systems Service (IERS), whose job it is to make sure the world's clocks are properly set. The IERS created the leap second in 1972 to keep atomic time aligned with solar time. Since then 25 leap seconds have been added to attomic clocks. The last leap second was added December 31, 2008.
At first, leap seconds weren't a problem. Computers accomodated them by setting their clocks backwards by one second at the end of the day.
But growth of the web, and the rise of global Internet companies, made synchronization more crucial. If a company's computers fell out of synch by a second, bad things could happen: email that arrived at the moment of a leap second could get lost, data that was supposed to be saved could be forgotten. Any number of unforeseen errors could occur—and did.
A leap second that was added in 2005 caused enough problems for Google that it invented a clever solution. Instead of adding the second all at once, it added milliseconds over the course of a day. "All of our servers were then able to continue as normal with the new year, blissfully unaware that a leap second had just occurred," Christopher Pascoe, one of Google's site reliability engineers, wrote in a blog post in September, which should have been required reading for any system administrator.
Instead, it was the sysadmins who remained blissfully unaware of the problems the leap second could cause, until it was too late.
"Spent the last two hours recovering servers," one engineer wrote on Saturday. "Tomorrow will be another interesting day."
