ExecutorService's surprising performance break-even point --- rules of thumb?

The Fixed ThreadPool's ultimate porpose is to reuse already created threads. So the performance gains are seen in the lack of the need to recreate a new thread every time a task is submitted. Hence the stop time must be taken inside the submitted task. Just with in the last statement of the run method.

In case it is useful to others, here are test results with a realistic scenario - use ExecutorService repeatedly until the end of all tasks - on a Samsung Android device.

 Simple computation (MS): 102
 Use threads (MS): 31049
 Use ExecutorService (MS): 257

Code:

   ExecutorService executorService = Executors.newFixedThreadPool(1);
        int count = 100000;

        //Simple computation
        Instant instant = Instant.now();
        for (int i = 0; i < count; i++) {
            double x = Math.random() * Math.random();
        }
        Duration duration = Duration.between(instant, Instant.now());
        Log.d("ExecutorPerformanceTest", "Simple computation (MS): " + duration.toMillis());


        //Use threads
        instant = Instant.now();
        for (int i = 0; i < count; i++) {
            new Thread(() -> {
                double x = Math.random() * Math.random();
            }
            ).start();
        }
        duration = Duration.between(instant, Instant.now());
        Log.d("ExecutorPerformanceTest", "Use threads (MS): " + duration.toMillis());


        //Use ExecutorService
        instant = Instant.now();
        for (int i = 0; i < count; i++) {
            executorService.execute(() -> {
                        double x = Math.random() * Math.random();
                    }
            );
        }
        duration = Duration.between(instant, Instant.now());
        Log.d("ExecutorPerformanceTest", "Use ExecutorService (MS): " + duration.toMillis());

I've faced a similar problem, but Math.random() was not the issue.
The problem is having many small tasks that take just a few milliseconds to complete. It is not much but a lot of small tasks in series ends up being a lot of time and I needed to parallelize.

So, the solution I found, and it might work for those of you facing this same problem: do not use any of the executor services. Instead create your own long living Threads and feed them tasks.

Here is an example, just as an idea don't try to copy paste it cause it probably won't work as I am using Kotlin and translating to Java in my head. The concept is what's important:

First, the Thread, a Thread that can execute a task and then continue there waiting for the next one:

public class Worker extends Thread {
  private Callable task;
  private Semaphore semaphore;
  private CountDownLatch latch;

  public Worker(Semaphore semaphore) {
    this.semaphore = semaphore;
  }

  public void run() {
    while (true) {
      semaphore.acquire(); // this will block, the while(true) won't go crazy
      if (task == null) continue;

      task.run();
      if (latch != null) latch.countDown();

      task = null;
    }
  }

  public void setTask(Callable task) {
    this.task = task;
  }

  public void setCountDownLatch(CountDownLatch latch) {
    this.latch = latch;
  }
}

There is two things here that need explanation:

• the Semaphore: gives you control over how many tasks and when they are executed by this thread
• the CountDownLatch: is the way to notify someone else that a task was completed

So this is how you would use this Worker, first just a simple example:

  Semaphore semaphore = new Semaphore(0); // initially the semaphore is closed
  Worker worker = new Worker(semaphore);
  worker.start();

  worker.setTask( .. your callable task .. );
  semaphore.release(); // this will allow one task to be processed by the worker

Now a more complicated example, with two Threads and waiting for both to complete using the CountDownLatch:

  Semaphore semaphore1 = new Semaphore(0);
  Worker worker1 = new Worker(semaphore1);
  worker1.start();

  Semaphore semaphore2 = new Semaphore(0);
  Worker worker2 = new Worker(semaphore2);
  worker2.start();

  // same countdown latch for both workers, with a counter of 2
  CountDownLatch countDownLatch = new CountDownLatch(2);
  worker1.setCountDownLatch(countDownLatch);
  worker2.setCountDownLatch(countDownLatch);

  worker1.setTask( .. your callable task .. );
  worker2.setTask( .. your callable task .. );
  semaphore1.release();
  semaphore2.release();

  countDownLatch.await(); // this will block until 2 tasks have been completed

And after that code runs you could just add more tasks to the same threads and reuse them. That's the whole point of this, reusing the threads instead of creating new ones.

It is unpolished as f*** but hopefully this gives you an idea. For me this was an improvement compared to no multi threading. And it was much much better than any executor service with any number of threads in the pool by far.