with this code i could get list of author and book title from first url!! how to crawl multiple urls data using beautifulsoup?

user_11808661 0
Viewed 341
import requests, bs4
import numpy as np
import requests
import pandas as pd
import requests
from bs4 import BeautifulSoup
from pandas import DataFrame


urls = ['http://www.gutenberg.org/ebooks/search/? 
sort_order=title','http://www.gutenberg.org/ebooks/search/?sort_order=title&start_index=26']
for url in urls:
    page = requests.get(url)
    soup = BeautifulSoup(page.content, 'html.parser')
    tb = soup.find_all('span', class_='cell content')
    soup_books = soup.findAll("span",{"class":"title"})  #books
    soup_authors= soup.findAll("span",{"class":"subtitle"}) #authors

    article_title = []
    article_author = []

    soup_title= soup.findAll("span",{"class":"title"})  # books
    soup_para= soup.findAll("span",{"class":"subtitle"})  #authors
for x in range(len(soup_para)):
    article_title.append(soup_title[x].text.strip())
    article_author.append(soup_para[x].text)

data = {'Article_Author':article_author, 'Article_Title':article_title}
df = DataFrame(data, columns = ['Article_Title','Article_Author'])
print(df)
len(df)

I need to crawl data from website 'http://www.gutenberg.org/ebooks/search/? sort_order=title' till the end of the page how can i iterate through the pages to get all the authors and titles of there work in that section

pandas web-scraping web-crawler beautifulsoup python
1 Answers

Do you mean after the first 25 results, you want to navigate to the next page and get the next page's results? You can use beatufiulsoup to get the URL of the "Next" button at the bottom right of the page:

next_url = soup.find('a', {'title': 'Go to the next page results.'})

and then run your code again with the new URL.

user_93669400
Related