Java Executor Service - Out Of Memory

user_3383301 0
Viewed 171

I am using the java executor service to transfer data from one DB to another.

ExecutorService executor = (ExecutorService) Executors.newFixedThreadPool(10);

Once executor service is initialized. Records are retrieved from Source DB in batches of 500 and those records are indexed into the target DB.

while(true){  
   List<?> docs = getDataFromSourceDB();
   if(docs.size > 0){
       IndexdataToTargetDBThread thread = new IndexdataToTargetDBThread(docs)
       executor.submit(thread);
   }else{
        break;
   }
} 

executor.shutdown()
while (!eService.isTerminated()) {
}

The function getDataFromSourceDB() returns 500 records on each call and if no records are available it will return an empty array. The class IndexdataToTargetDBThread implements Runnable. It indexes the given documents to the target DB.

The exception Out Of Memory occurs when the above function is run. The Source DB has close to 1M records, hence the while loop is processed around 2000 times. After processing around 700 times out of memory occurs and entire DB transfer process fails. Is there any efficient way to handle this?

executorservice multithreading java
1 Answers

I see two problems. First, it is an unlimited data size fetched from DB. You may restrict the number of records that exist at the same time in the application. Create buffer(It could be a queue with limited elements) and flush batch once it ready to be wrote: https://docs.oracle.com/javase/8/docs/api/java/util/Queue.html)

Second: not sure about the database's isolation level. The next article describes some issues with concurrent insert in PostgreSQL: https://www.postgresql.org/files/developer/concurrency.pdf

Also, you can try to increase the java memory limit via -Xmx .. option: Increase heap size in Java

But that not the best way :)

user_136257920
Related