Getting Google search results with JSoup

Viewed 774

I'm trying to get a list of Google search results via JSoup. The method I'm currently using works perfectly fine for the first page (n), but for n+1 pages it does not work so well. Here is how I'm getting the first page:

doc = Jsoup.connect(search).userAgent("Chrome").get();
links = doc.getElementsByClass("r");

The search string would contain something like: https://www.google.com/search?q=apple for the first page. Then my code looks like this for n+1 pages:

for(int i = 1; i <= pages; i++){
    search = "https://www.google.com/#q=" + keyword + "&start=" + (i*10);
    doc = Jsoup.connect(search).userAgent("Mozilla").get();
    links.addAll(doc.getElementsByClass("r"));
}

The search for n+1 pages would look something like:https://www.google.com/#q=apple&start=10. The main issue that I'm having is that doc.getElementsByClass("r") in the n+1 searches contains no elements. This means that the class r does not exist in what JSoup returns. I verified this by searching through doc.toString(). Does anyone have any suggestions?

Thank you!

1 Answers
Related