Using a disk backed queue

Question

Under certain conditions, one of our servers running legacy code in a Wildfly application server, is suffering thread starvation and needs to be restarted.

After an arduous investigation, I stumbled upon this silly code:

private void addToQueue (Item e) {

     if (!_queue.offer(e, 200L, TimeUnit.MILLISECONDS)) { 
              ThreadService.getInstance().schedule("retry process", () -> {
                addToQueue(e);
                return null;
              }, 5L, TimeUnit.SECONDS);
            }
    }

ThreadService is the Wildfly implementation of the Java SE Executor Service, that provides a limited number of threads (16).

Sometimes, during reconnections between services, we receive in short time a huge amount of items to be processed (~100k), and as you can see it spams the ThreadService with scheduled tasks.

An obvious solution would be to increase the queue, which has currently a capacity of 20k. However I am afraid this would end up in other problems. Obviously this thread creation spamming needs to be eliminated.

Since processing these items is a non critical task I was thinking on using a disk backed queue, so it can be done in a separated process at slow pace.

Searching a bit, I have seen this project: Tape by Square

I would like to know your opinion about this solution, which kind of reminds me a bit to the pipes in Linux I used years ago. What do you think?

Using a disk backed queue

0 Answers