How to Improve the processing of files using executor framework in java?

user_13460350 0
Viewed 123

I've to process the lot of files which will be residing into some local directory of system, those files need to process and there will be a single entry called 'reqId', so for each reqId i have to call Rest Api with Post request. First I am fetching the files one after another and creating the list of reqId and then passing the list to ReqExecutorService to process further.

 public static void main( String[] args )
    {
        String srcFolder=config.getProperty(Constants.REQ_ID_FILE_PATH);
        Optional<String[]> fNames =FileUtils.getFileNames(srcFolder);

        if(fNames.isPresent()) {
            for(String fileName:fNames.get()) {
                List<String> list = null;
                try (Stream<String> stream = Files.lines(Paths.get(srcFolder + "/" + fileName))) {


                    list = stream.collect(Collectors.toList());

                } catch (IOException e) {
                    LOGGER.error(String.format("Exception occurred while loading the file from %s",srcFolder));
                }

                //list.forEach(System.out::println);
                ReqExecutorService.execute(list);

            }

        }

In the below method, I am using the concurrency to process each list by the multiple threads(no. can be adjusted ) 

public static void execute(List<String> reqList) {
        LOGGER.info(String.format("RequestExecutor is processing the req's of size:: %s",reqList.size()));
        ExecutorService exService=Executors.newFixedThreadPool(6);
        Instant startTime = Instant.now();
        reqList.parallelStream().forEach(reqId->exService.execute(new ReqRunnable(reqId)));
        Instant endTime = Instant.now();
        long timeElapsed = Duration.between(startTime, endTime).toMillis();
        LOGGER.info("Time took to complete  "+timeElapsed);
        exService.shutdown();
        try {
            exService.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);

        }catch(InterruptedException ee) {
             LOGGER.warn(String.format("Thread %s has been interuptted %s ",Thread.currentThread().getName(),ee));
                // Restore interrupted state...
                Thread.currentThread().interrupt();

        }
    }

Rest Api code will go here!

public class ReqRunnable implements Runnable {
private static final Logger LOGGER=LoggerFactory.getLogger(ReqRunnable.class);
    String reqId;
     ReqRunnable(String reqId)
    {
        this.reqId=reqId;
    }
    @Override
    public void run() {
        //restApi code to post request with reqId
    }

}

What can possibly be done here to improve the performance of the system because there will be one million files and each file could have one million lines?

Output : First two list taking much time to processing even though others have the same size ??

2020-05-22 04:19:44 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 28 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 26 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 7 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 5247
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 3 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 7 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 7 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 6384
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 3 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 702
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 1 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 4 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 3 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 702
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 1 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 3946
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 5247
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 3 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 702
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 6 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 7 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 702
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 6384
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 3 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 5247
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 1 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 6384
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 1 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 6384
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 1 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 6384
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 4 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 16430
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 26795
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 4 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 5247
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 1 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 26795
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 4 millis
2020-05-22 04:19:45 INFO  ReqExecutorService:18 - RequestExecutor is processing the req's of size:: 26795
2020-05-22 04:19:45 INFO  ReqExecutorService:24 - Time took to complete 2 millis
executorservice file java performance
0 Answers
Related