How to hit 1000 endpoints using multi threading in groovy?

Admin 1
Viewed 5593

I have a requirement to hit endpoints more than 1000 times to fetch some data from website. So i read some tutorials to use Multi Threading to achieving it. But at a time i want to use only 13 threads on same method.

So basically i am using ExecutorService to run 13 threads at one time: 

ExecutorService threadPool = Executors.newFixedThreadPool(13);
for (int itLocation = 0; itLocation < locationList.size(); itLocation++) {
  //some code like
  ScraperService obj = new ScraperService(threadName,url)
  threadPool.submit(obj);
}
threadPool.shutdown();

My Groovy Class named as ScraperService is implementing the Runnable interface.

@Override
        void run() {
            println("In run method...................")
            try {
                Thread.sleep(5000);
                someMethod()
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }

Problem:

My problem is that my ExecutorService.submit(obj) and ExecutorService.execute(obj) is not calling my run() method of Runnable interface.

In Groovy/Grails:

There is also a executor plugin Executor Plugin in grails but i didn't found any appropriate example how to use it.

executorservice threadpool grails multithreading groovy
3 Answers

GPars is excellent for this type of thing.

Your ScraperService can just be responsible for handling the scraped data like below or maybe fetching it too, whatever.

import groovyx.gpars.GParsPool

def theEndpoint = 'http://www.bbc.co.uk'

def scraperService

GParsPool.withPool( 13 ) {
    (1..1000).eachParallel {
        scraperService.scrape theEndpoint.toURL().text
    }
}
user_65440830

the threadPool.submit does not execute task

use threadPool.execute(obj) or threadPool.submit(obj).get()

instead of threadPool.submit(obj)

check the documentation for details:

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html#submit(java.util.concurrent.Callable)

example:

import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors


ExecutorService pool=Executors.newFixedThreadPool(3)

for(int i=0;i<7;i++){
    int x=i;
    Thread.sleep(444);
    pool.execute{
        println "start $x"
        Thread.sleep(3000+x*100);
        println "end   $x"
    }
}

println "done cycle"
pool.shutdown()                 //all tasks submitted
while (!pool.isTerminated()){}  //waitfor termination
println 'Finished all threads'
user_12766640

First of all i think there is problem in groovy service class with @Transnational annotation which doesn't allow to call run() method of Runnable interface. If you'll remove the @Transnational then it will call the run() method. It happened also in my case. But i am not sure, there may be some another reason. You can directly use:

ExecutorService threadPool =  Executors.newFixedThreadPool(13)
threadPool.execute(new Runnable() {
                            @Override
                            void run() {
                                Thread.sleep(5000);
                                someMethod()
                            }
                        })

Extra (As i read your question)

If you are using multiple threads on same method then it can be complex as all threads will use the same local variables of that method which can occurs problems. It is better to use multiple threads for different different work.

But if you want to use same method for executing multiple threads then in my scenario it is better to use Executors.newSingleThreadExecutor().

ExecutorService threadPool = Executors.newSingleThreadExecutor();

newSingleThreadExecutor() calls a single thread so if you want to execute multiple tasks on it then it does not create multiple threads. Instead, it waits for one task to complete before starting the next task on the same thread.

Speed: newSingleThreadExecutor in comparison of multi threads will be slow but safer to use.

user_50682130
Related