ExecutorService slows down , bogs down my pc

Viewed 588

I am writing a parser for a website , it has many pages (I call them IndexPages) . Each page has a lot of links (about 300 to 400 links in an IndexPage). I use Java's ExecutorService to invoke 12 Callables concurrently in one IndexPage. Each Callable just fire a http request to one link and do some parsing and db storing actions. When first IndexPage finished , program progresses to second IndexPage , until no next IndexPage found.

parsing 1st page When running , it seems OK , I can observe the threads working/scheduling well. Each link's parsing/storing just takes about 1 to 2 seconds.

After running 2 hours But as time goes by , I observed each Callable(parsing/storing) takes longer and longer. Take this picture for example , sometimes it takes 10 or more seconds to finish a Callable (The green bar is RUNNING , the purple bar is WAITING). And my PC is bogging down , everything becomes sluggish.

This is my main algorithm :

ExecutorService executorService = Executors.newFixedThreadPool(12);    
String indexUrl = // Set initial (1st page) IndexPage
while(true)
{
  String nextPage = // parse next page in the indexUrl

  Set<Callable<Void>> callables = new HashSet<>();
  for(String url : getUrls(indexUrl))
  {
    Callable callable = new ParserCallable(url , … and some DAOs);
    callables.add(callable);
  } 

  try {
    executorService.invokeAll(callables);
  } catch (InterruptedException e) {
    e.printStackTrace();
  }

  if (nextPage == null) 
    break;

  indexUrl = nextPage;
} // true
executorService.shutdown();

The algorithm is simple and self-explanatory. I wonder what may cause such situation ? Anyway to prevent such performance degradation ?

CPU/Memory The CPU/Memory/Heap shows reasonable usage.

Environments Environments , FYI.

==================== updated ====================

I've change my implementations from ExecutorService to ForkJoinPool :

ForkJoinPool pool=new ForkJoinPool(12);
String indexUrl = // Set initial (1st page) IndexPage
while(true)
{
  Set<Callable<Void>> callables = new HashSet<>();
  for(String url : for(String url : getUrls(indexUrl)))
  {
    Callable callable = new ParserCallable(url , DAOs...);
    callables.add(callable);
  }
  pool.invokeAll(callables);

  String nextPage = // parse next page in this indexUrl
  if (nextPage == null)
    break;

  indexUrl = nextPage;
} // true

It takes longer than ExecutorService's solution. ExecutorService takes about 2 hours to finish all pages , while ForkJoinPool takes 3 hours , and each Callable still takes longer and longer time to complete (from 1 sec to 5,6 or even 10 seconds). I don't mind it takes longer , I just hope it takes constant time (not longer and longer) to finish a job .

I am wondering if I create a lot of (non-thread-safe) GregorianCalendar , Date and SimpleDateFormat objects in the parser and cause some thread issue. But I didn't reuse these objects or pass them among threads. So I still cannot find the reason.

3 Answers
Related