I'm building a web crawler. I basically have 2 seperate methods, crawl and scrape. Scrape queries and scrapes the relevant details from the URL, crawl calls scrape and checks that the output doesn't already exist in a list, checks the depth hasn't been exceeded, and if any new URLS exist in the scraped URLs, it will recursively call crawl on those URLs.
I want to perform the crawling asynchronously (ie, call initial crawl, get back first set of scraped URLS, then crawl all the URLS from that page in parallel, then the URLs from there in parallel, etc etc.) Is there a way to implement CompletableFutures with the recursive approach, or does it need to be totally reworked?