This is the code. It scratch a forum post and display the results. It should return only 1 post but it returns 19 posts. Thread argument is a url of forum. First page and second page is a range to scrape.
@PostMapping("/search")
public String scrapPosts(@RequestParam(name = "thread", required = true) final String thread,
@RequestParam(name = "firstPage", required = true)final int firstPage,
@RequestParam(name = "secondPage", required = true)final int secondPage,
Model model) throws IOException {
ExecutorService executorService = Executors.newFixedThreadPool(20);
List<Post> postList = new ArrayList<>();
for (int i = firstPage; i <secondPage; i++) {
int finalI = i;
Document doc = Jsoup.connect(thread + "page-" + finalI).get();
Elements posts = doc.getElementsByClass("message-inner");
posts.remove(posts.size() - 1);
executorService.submit(()->{
try {
for (int j = 0; j < posts.size(); j++) {
int finalJ = j;
postList.add(createNewPost(posts.get(finalI), thread, finalJ));
}
} catch (Exception e) {
e.printStackTrace();
}
});
}
executorService.shutdown();
try {
executorService.awaitTermination(100000, TimeUnit.SECONDS);
System.out.println("Srednia cena to \n");
model.addAttribute("postList", postList);
} catch (InterruptedException e) {
System.out.println("Could not stop in alloted time");
}
executorService.shutdown();
return "search";
}
And the return is something like this:
