I am reading a list of pdf files from DB and parsing them and performing some tasks with them.
When I am reading this list of pdf's , I have seen extracting images from PDF is taking more time and I don't to block my main thread for reading images. So I want to execute extracting images in a separate thread.
I want to read images from one pdf after another, without loading all the pdfs into memory at once (due to memory concern). So I just want 2 threads; one should be main thread (which reads some text from pdf and does some other stuff) and other should be a thread which extracts images and return set of image objects.
One caveat here is that the images from pdfs can be same in their content , so I want to remove duplicate images using checksum or some other means before collecting their result.
I don't want to hold images in memory until all tasks are getting completed, I want to remove duplicates as on when I get the result of one pdf
So the real question is , I need to submit multiple tasks to a thread pool of size 1 and need to remove duplicates as on when I get the result so that I don't need to hold the image in memory for longer time.
Below is the idea of what have I tried.
I have removed unnecessary things from the code like Images and its content and converted code to String based problem.
public static void main(String[] args) throws InterruptedException, ExecutionException {
Map<String, Integer> uniqueImages = new HashMap<>();
ExecutorService newFixedThreadPool = Executors.newFixedThreadPool(1);
List<CompletableFuture<String>> futureList = new ArrayList<>();
for(int i = 0; i<20000; i++) {
CompletableFuture<String> obj = CompletableFuture.supplyAsync(()->{
//Assume lot of duplicates
return UUID.randomUUID().toString();
}, newFixedThreadPool).thenApply((x)->{
if(uniqueImages.containsKey(x)) {
int val = uniqueImages.get(x);
uniqueImages.put(x, val+1);
}
else {
uniqueImages.put(x, 1);
}
return x;
});
futureList.add(obj);
}
for(CompletableFuture<String> future: futureList) {
future.get();
}
System.out.println(uniqueImages.size());
}
I'm worried if this code really works or throws ConcurrentModification Exception.
- Does
uniqueImagesmap really contains unique images and their counts? - Are there any hidden issues?
- Is there better way to solve my use case?