JIRA Performance Problems due to System Time Settings
You are running a flavor of Linux and Java and observe one or both of the following to happen.
- CPU load spikes to a consistent 100%
- The application will not restart and stop at the plugins
The following possibly appears in the
atlassian-jira.log dependent on how your server reacts:
2012-06-30 08:37:32,480 jira-monitoring-plugin ERROR [jira.plugins.monitor.MetricsCollectorTask] Error running task java.lang.IllegalArgumentException: Bad sample time: 1341063452. Last update time was 1341075248, at least one second step is required at org.rrd4j.core.RrdDb.store(RrdDb.java:553) at org.rrd4j.core.Sample.update(Sample.java:197) at com.atlassian.jira.plugins.monitor.rrd4j.RrdUpdater.addSample(RrdUpdater.java:76) at com.atlassian.jira.plugins.monitor.MetricsCollectorTask.doRun(MetricsCollectorTask.java:41) at com.atlassian.jira.plugins.monitor.MetricsCollectorTask.run(MetricsCollectorTask.java:30) at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source)
The effects of this issue may not be limited to the ones described above. Your service may also be impacted through interim elements (proxy-, database servers for example)
The potential cause of this is from the leap second bug that starting affecting servers on July 1 and December 31 as covered on this Wired article. Another cause is the system clock may be losing time and is not synchronising with a NTP server.
Run the following command on the server:
/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date
You may have to stop the ntpd service instead of ntp. This will be dependent on the flavor of linux you are running.
If you are running on a virtual machine, check your host load to see if your other virtual machines are creating an increased amount of load. You'll want to apply the fix on those servers as well in order to return normality to your system. If other virtual machines are not addressed, your plugins start up could reach the timeout limit and will not start correctly.
See our public blog post on the topic: What Atlassian application administrators need to know about the “leap second” bug lurking in your systems