If you're building a small parser, there is no need to use a dedicated web scraping framework, such as Scrapy. You can use Beautiful Soup library instead. This library is almost as fast as scrapy and does not require additional settings ie. settings.py file.
To scrape a Google search with pagination, one of the ways is to use the start URL parameter which is equal to 0 by default. 0 means the first page, 10 is for the second, and so on. Or you can use SerpApi pagination for Google Search results which is an API.
# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
"q": "Elon Musk",
"hl": "en", # language
"gl": "us", # country of the search, US -> USA
"start": 0, # number page by default up to 0
"filter": 0 # shows more than 10 pages. By default up to ~10-15 if filter = 1.
}
Also, default search results return about ~10-15 pages. To increase the number of returned pages, you need to set the filter parameter to 0 and pass it to the URL which will return 10+ pages. Basically, this parameter defines the filters for Similar Results and Omitted Results.
There are snippets in which there is no description, so a corresponding try/except check has been added:
try:
description = f'Description: {result.select_one(".VwiC3b").text}'
except:
description = None
While the next button exists, you need to increment the ["start"] parameter value by 10 to access the next page if it's present, otherwise we need to break out of the while loop:
if soup.select_one('.d6cvqb a[id=pnnext]'):
params["start"] += 10
else:
break
Also, make sure you're using request headers user-agent to act as a "real" user visit. Because default requests user-agent is python-requests and websites understand that it's most likely a script that sends a request. Check what's your user-agent.
Code and full example in online IDE:
from bs4 import BeautifulSoup
import requests, lxml
# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
"q": "Elon Musk",
"hl": "en", # language
"gl": "us", # country of the search, US -> USA
"start": 0, # number page by default up to 0
"filter": 0 # show all pages by default up to 10
}
# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}
page_num = 0
while True:
page_num += 1
print(f"{page_num} page:")
html = requests.get("https://www.google.com/search", params=params, headers=headers, timeout=30)
soup = BeautifulSoup(html.text, 'lxml')
for result in soup.select(".tF2Cxc"):
title = f'Title: {result.select_one("h3").text}'
link = f'Link: {result.select_one("a")["href"]}'
try:
description = f'Description: {result.select_one(".VwiC3b").text}'
except:
description = None
print(title, link, description, sep="\n", end="\n\n")
if soup.select_one('.d6cvqb a[id=pnnext]'):
params["start"] += 10
else:
break
Output:
1 page:
Title: Elon Musk - Wikipedia
Link: https://en.wikipedia.org/wiki/Elon_Musk
Description: Elon Reeve Musk FRS is a business magnate and investor. He is the founder, CEO, and Chief Engineer at SpaceX; angel investor, CEO, and Product Architect of ...
Title: Elon Musk - Forbes
Link: https://www.forbes.com/profile/elon-musk/
Description: Elon Musk is working to revolutionize transportation both on Earth, through electric car maker Tesla -- and in space, via rocket producer SpaceX.
Title: Elon Musk - Tesla
Link: https://www.tesla.com/elon-musk
Description: As the co-founder and CEO of Tesla, Elon leads all product design, engineering and global manufacturing of the company's electric vehicles, battery products and ...
... other results from the 1st and subsequent pages.
10 page:
Title: Elon Musk: Revolutionary private space entrepreneur
Link: https://www.space.com/18849-elon-musk.html
Description: Oct 30, 2021 — As the CEO of SpaceX and Tesla, Elon Musk is transforming transport in space and on Earth.
Title: Elon Musk: When I saw the Tesla CEO for who he really is.
Link: https://slate.com/technology/2022/05/elon-musk-tesla-twitter-fables.html
Description: May 27, 2022 — Overtaken by curiosity, I had decided to spend a long Memorial Day weekend in California's Central Valley to see if Elon Musk's latest bit ...
Title: Elon Musk on 'Giant Money Furnaces' and Not Letting Tesla ...
Link: https://www.barrons.com/articles/elon-musk-tesla-bankruptcy-twitter-51655982269
Description: Jun 23, 2022 — Tesla CEO Elon Musk spoke with a Tesla owners club. He talked bankruptcy, advertising, full self driving, and birthrates.
... other results from 10th page.
