BS4 Python: Trying to take page links from Google but the URLs I get are all the same

Question

I have made a python script with BS4 to try to get the results from a google search.

Problem: I can only take data from Page 1 of Google

Attempt to Solve: I tried to extract each pages href at the bottom of google page list and get the href for each page from 1,2,3...10 and repeat my process as I did for page 1.

Problem to attempt: When I tried to extract pages 1..10 URL those page links are not the same as the ones in the inspect element on google.com . google inspect element links

import requests
from bs4 import BeautifulSoup
import functions
#-----------------------------------------------------------------------
url = 'https://google.com/search?q=manga'  # main link to get data
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'}  # headers
source = requests.get(url,headers = headers).text  # url source

#making tasty soup
soup = BeautifulSoup(source, 'lxml')
#-----------------------------------------------------------------------

pages = []


search_div = soup.find_all(class_='rc')  # find all divs    tha contains search result

def get_result(search):
    result = []
    for result in search:  # loop result list
        print('Title: %s' % result.h3.string)  # geting h3
        result.append(result.h3.string)

        print('Url: %s' % result.a.get('href'))  # geting a.href

        print('Description: %s' % result.find(class_='st').text)  # description
        print('
###############
')

    return result
result = get_result(search_div)

a = soup.find('table')
b = soup.find("tr", {'valign':'top'})

for i in b:
    print(str(i))

edit: the code above produces:

Title: Manga - Wikipedia
Url: https://en.wikipedia.org/wiki/Manga
Description: Manga are comics or graphic novels originating from Japan. Most manga conform to a style developed in Japan in the late 19th century, though the art form has ...

###############

Title: Read the Best Manga - VIZ
Url: https://www.viz.com/read
Description: Action, adventure, fantasy, mystery, romance and more—thousands of manga volumes for every fan!

###############

Title: Manga Toon - Free manga, comic and novel reader online
Url: https://mangatoon.mobi/
Description: MangaToon is a Global APP for Reading Comic Manga and Novel. Different comics in Action, Romance, Boys' love, Comedy, Horror and more are updated ...

###############

Title: Read Popular Manga Online - Crunchyroll
Url: https://www.crunchyroll.com/comics/manga
Description: Read your favorite Japanese manga online on Crunchyroll including Attack on Titan, Fairy Tail, The Seven Deadly Sins, Fuuka, Knight's & Magic, and more.

###############

Title: Manga Books - Goodreads
Url: https://www.goodreads.com/genres/manga
Description: Manga. Japanese or Japanese-influenced comics and graphic novels. Usually printed in black-and-white. There are many genres inside manga, the most distinct being shojo (for girls) and shonen (for boys).

###############

Title: Manga and Anime Books | Barnes & Noble®
Url: https://www.barnesandnoble.com/b/books/graphic-novels-comics/manga/_/N-29Z8q8Zucc
Description: Discover an extensive collection of manga and anime books at Barnes & Noble. Shop a wide variety of Manga series, boxed sets, bestsellers, and more.

###############

Title: 50 Best Manga You Must Read Right Now: Classics And New ...
Url: https://bookriot.com/2020/05/26/best-manga/
Description: May 26, 2020 - New to reading manga and don't know where to start? Want to find a new series to dive into? Here's a list of the 50 best manga to add to your ...

###############

Title: MANGA Plus
Url: https://mangaplus.shueisha.co.jp/updates
Description: "MANGA Plus by SHUEISHA" is the official manga reader from Shueisha Inc., and is available globally. We publish the greatest manga in the world such as ...

###############


1
2
3
4
5
6
7
8
9
10
Next

Process finished with exit code 0

How I tested if the links were correct: I went into my search bar on chrome and entered "google.com" to see if the page number changed. I tried each link and I am always on the first page.

user_10035985 · Answer

This script prints all headers from google page and then tries to find next link (link that contains word "Next"). If the link is found, continue loading the next page:

import requests
from bs4 import BeautifulSoup


url = 'https://google.com/search?q=manga&hl=en'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

page = 1
while True:
    print()
    print('Page {}...'.format(page))
    print('-' * 80)

    soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')
    for h in soup.select('h3'):
        print(h.get_text(strip=True))

    next_link = soup.select_one('a:contains("Next")')
    if not next_link:
        break

    url = 'https://google.com' + next_link['href']
    page += 1

Prints:

Page 1...
--------------------------------------------------------------------------------
Manga - Wikipedia
Read the Best Manga - VIZ
Manga Toon - Free manga, comic and novel reader online
Read Popular Manga Online - Crunchyroll
Manga Books - Goodreads
MANGA Plus
Videos
MANGA Plus by SHUEISHA - Apps on Google Play
MANGA.CLUB｜Read Free Official Manga Online!
50 Best Manga You Must Read Right Now: Classics And New ...
Images

Page 2...
--------------------------------------------------------------------------------
Top 10 Best Manga Series - YouTube
What are Manga and Anime - MIT
manga - Wiktionary
15 Best Manga Of All Time (According To MyAnimeList) | CBR
Images for manga
Guided Search Filters
Manga Storm on the App Store
A Beginner's Guide to Manga | The New York Public Library
Manga and Anime Books | Barnes & Noble®
Manga | Definition of Manga by Merriam-Webster
Manga Rock - Online Manga Reader

Page 3...
--------------------------------------------------------------------------------
manga - Urban Dictionary

...and so on.

BS4 Python: Trying to take page links from Google but the URLs I get are all the same

1 Answers