Java time getting out of sync

user_1262024 0
Viewed 1227

Recently, I am experiencing a very strange error which I am unable to reproduce locally. It happens on a server (CentOS) inside a Spring Boot application.

calling new Date() (java.util.Date) produces wrong time. And I am not talking about an hour difference due to different timezone. The datetime is off by X minutes. It seems that it is gradually starting to lack behind the current time.

When getting the current system clock time, it seems to be correct but the Java time is gradually increasing the difference (not by one minute every minute but slower). Almost like the JVM existed in some time bubble with different laws of time.

This issue starting happening accidentally today after a long run without any problems.

Could someone suggest what should I try to debug this issue? I am clueless and cannot replicate this issue locally (everything works on local machine).

EDIT: The server is running on a virtualized VM stack, as I just found out. I am not sure about the specific HW configuration, though.

EDIT2: I think I found the problematic part. The component causing this problem is running inside a threadpool, so I suspect that all threads in the pool are busy and the requests are queued so it takes gradually more time to handle the requests in which I am setting current timestamp.

Example:

@Async
public Date sendTimeToOtherServer() {
  Date date = new Date()
  // code that sends the date to different server
}
spring-boot time java
1 Answers

Java gets the time from the OS via a syscall. This should in turn get the time from a hardware clock.

Now hardware clocks can drift. When you are running a virtual machine, the hardware clock may be emulated by the virtual framework, and may drift more than you would expect. This may depend on the load on your VM ... or load on the hypervisor from your VM and others on the compute hardware.

But the solution is likely to be to use NTP utilities to sync the VM's system clock with a reliable off-machine time service.

I increased the thread pool size and it started acting correctly so it must be this bottleneck in the thread pool

You might be prematurely jumping to conclusions here. FWIW, I have never heard of clock accuracy being dependent on the size of a Java thread pool.

What I meant is, that it probably took some time for the thread pool to be ready to provide some thread to handle my task in which I was calling new Date().

Ah. I see. Well if you are capturing the request start time in a worker thread and there are insufficient threads in the pool, then what you are getting with new Date() is most likely an accurate timestamp (relative to the system clock); i.e. time is NOT out of sync.

But the question is whether it is a good thing to increase the thread pool size in this situation. Are the delayed request start times due to having too few workers, or to having a too high request rate?

  • In the former case, increasing the pool size is the right thing to do.

  • In the latter case, it won't help much, and may make things worse.

The ultimate limiting factor on throughput will be factors like the number of cores, the memory size, database performance, network bandwidth and so on. Increasing the thread pool may allow more requests to be processed in parallel. However, if you are already resource starved, this won't increase the request processing rate. Indeed, an excessive number of worker threads can lead to:

  • increased memory usage and GC load, and
  • individual requests taking so long that the client times them out.
user_1399850
Related