I am trying to scrape tables from this website https://securities.stanford.edu/filings.html?page=1.0. After browsing through videos and posts here, I thought one of the answers How to scrape paginated table with BeautifulSoup and store results in csv? might be applicable. However, for the line soup.select('li.active + li a[href]'), the return I get is
[<a href="resources.html">Resources</a>,
<a href="current-trends.html">Current Trends</a>,
<a data-toggle="modal" href="#myModalInvite">2</a>]
I am not sure why I am not getting the next page link. Why doesn't it return <a href="?page2.0"? I am new to webscrapping and I am stuck at this stage. Could someone help me with this? Thanks.
from bs4 import BeautifulSoup
import requests
import re
import pandas as pd
first_url = 'https://securities.stanford.edu/filings.html?page=1.0'
while True:
url = first_url
data = req.get(url).text
bs = BeautifulSoup(data,'html.parser')
data.append(pd.read_html(soup.select_one('table').prettify())[0])
soup.select('li.active + li a[href]')
# if soup.select_one('li.active + li a[href]'):
# url = soup.select_one('li.active + li a')['href']
# else:
# break
So, after getting cookies and headers for logging into the site before scrapping, I have the following code. Unfortunately, for some reason, it is only returning me the third page.
cookies = #cookies here
headers = #headers here
root_url = 'https://securities.stanford.edu/filings.html'
response = requests.get(url, cookies=cookies, headers=headers).text
url_list = []
data = []
for i in range(1,4):
url = root_url + '?page{}'.format(i)
print(url)
response = requests.get(url, cookies=cookies, headers=headers).text
bs = BeautifulSoup(response, 'html.parser')
data.append(pd.read_html(bs.select_one('table').prettify())[0])