Java Performance Testing

The one flaw in that approach is that the "real" time doSomething() takes to execute can vary wildly depending on what other programs are running on the system and what its load is. This makes the performance measurement somewhat imprecise.

One more accurate way of tracking the time it takes to execute code, assuming the code is single-threaded, is to look at the CPU time consumed by the thread during the call. You can do this with the JMX classes; in particular, with ThreadMXBean. You can retrieve an instance of ThreadMXBean from java.lang.management.ManagementFactory, and, if your platform supports it (most do), use the getCurrentThreadCpuTime method in place of System.currentTimeMillis to do a similar test. Bear in mind that getCurrentThreadCpuTime reports time in nanoseconds, not milliseconds.

Here's a sample (Scala) method that could be used to perform a measurement:

def measureCpuTime(f: => Unit): java.time.Duration = {

  import java.lang.management.ManagementFactory.getThreadMXBean
  if (!getThreadMXBean.isThreadCpuTimeSupported)
    throw new UnsupportedOperationException(
      "JVM does not support measuring thread CPU-time")

  var finalCpuTime: Option[Long] = None
  val thread = new Thread {
    override def run(): Unit = {
      f
      finalCpuTime = Some(getThreadMXBean.getThreadCpuTime(
        Thread.currentThread.getId))
    }
  }
  thread.start()

  while (finalCpuTime.isEmpty && thread.isAlive) {
    Thread.sleep(100)
  }

  java.time.Duration.ofNanos(finalCpuTime.getOrElse {
    throw new Exception("Operation never returned, and the thread is dead " +
      "(perhaps an unhandled exception occurred)")
  })
}

(Feel free to translate the above to Java!)

This strategy isn't perfect, but it's less subject to variations in system load.

The code shown in the question is not a good performance measuring code:

1. The compiler might choose to optimize your code by reordering statements. Yes, it can do that. That means your entire test might fail. It can even choose to inline the method under test and reorder the measuring statements into the now-inlined code.

2. The hotspot might choose to reorder your statements, inline code, cache results, delay execution...

3. Even assuming the compiler/hotspot didn't trick you, what you measure is "wall time". What you should be measuring is CPU time (unless you use OS resources and want to include these as well or you measure lock contestation in a multi-threaded environment).

The solution? Use a real profiler. There are plenty around, both free profilers and demos / time-locked trials of commercials strength ones.

Using a Java Profiler is the best option and it will give you all the insight that you need into the code. viz Response Times, Thread CallTraces, Memory Utilisations, etc

I will suggest you JENSOR, an open source Java Profiler, for its ease-of-use and no overheads on CPU. You can download it, instrument the code and will get all the info you need about your code.

You can download it from: http://jensor.sourceforge.net/

Keep in mind that the resolution of System.currentTimeMillis() varies between different operating systems. I believe Windows is around 15 msec. So if your doSomething() runs faster than the time resolution, you'll get a delta of 0. You could run doSomething() in a loop multiple times, but then the JVM may optimize it.

Have you looked at the profiling tools in netbeans and eclipse. These tools give you a better handle on what is REALLY taking up all the time in your code. I have found problems that I did not realize by using these tools.

I'd imagine you'd want to doSomething() before you start timing too, so that the code is JITted and "warmed up".

Well that is just one part of performance testing. Depending on the thing you are testing you may have to look at heap size, thread count, network traffic or a whole host of other things. Otherwise I use that technique for simple things that I just want to see how long they take to run.

That's good when you are comparing one implementation to another or trying to find a slow part in your code (although it can be tedious). It's a really good technique to know and you'll probably use it more than any other, but be familiar with a profiling tool as well.

Japex may be useful to you, either as a way to quickly create benchmarks, or as a way to study benchmarking issues in Java through the source code.