I'm trying to get a list of Google search results via JSoup. The method I'm currently using works perfectly fine for the first page (n), but for n+1 pages it does not work so well. Here is how I'm getting the first page:
doc = Jsoup.connect(search).userAgent("Chrome").get();
links = doc.getElementsByClass("r");
The search string would contain something like: https://www.google.com/search?q=apple for the first page. Then my code looks like this for n+1 pages:
for(int i = 1; i <= pages; i++){
search = "https://www.google.com/#q=" + keyword + "&start=" + (i*10);
doc = Jsoup.connect(search).userAgent("Mozilla").get();
links.addAll(doc.getElementsByClass("r"));
}
The search for n+1 pages would look something like:https://www.google.com/#q=apple&start=10.
The main issue that I'm having is that doc.getElementsByClass("r") in the n+1 searches contains no elements. This means that the class r does not exist in what JSoup returns. I verified this by searching through doc.toString(). Does anyone have any suggestions?
Thank you!