I am a beginner to crawling. I have a requirement to fetch the posts and comments from a link. I want to automate this process. I considered using webcrawler and jsoup for this but was told that webcrawlers are mostly used for websites with greater depth.
Sample for a page: Jive community website
For this page, when I view the source of the page, I can see only the post and not the comments. Think this is because comments are fetched through an AJAX call to the server.
Hence, when I use jsoup, it doesn't fetch the comments.
So how can I automate the process of fetching posts and comments?