‘Leap Second’ Bug Wreaks Havoc Across Java Production Systems

Filed under: — Aviran Mordo

On Saturday, at midnight Greenwich Mean Time, as June turned into July, the Earth’s official time keepers held their clocks back by a single second in order to keep them in sync with the planet’s daily rotation, and due to many production problems we were faceing here at Wix.com with our Java systems, and also according to reports from across the web, some of the net’s fundamental software platforms — including the Linux operating system and the Java application platform — were unable to cope with the extra second.

Many computing systems use what’s called the Network Time Protocol, or NTP, to keep themselves in sync with the world’s atomic clocks, and when an extra second is added, some just don’t know how to handle it.

We had problems with Hadoop, Tomcat, Jetty, Flum, Tyrent, Tokyo and other systems. One major symptoms we saw was system crushing and CPU overload.

Fortunately we found a solution which is simply to run the following linux command:

date; sudo date `date +"%m%d%H%M%C%y.%S"`; date;

This should solve your problem and you don’t have to restart your java applications

Powered by WordPress