How to scrape stock data incorporating pagination next tag using python bs4?

user_2424253 0
Viewed 105

The code can not get the next page, it only repeats in an infinite loop. I am using the example from oxylabs

Could you tell me what I'm doing wrong? Thank you.

import requests
from bs4 import BeautifulSoup as bs
from urllib.parse import urljoin

url = 'https://hnx.vn/en-gb/cophieu-etfs/chung-khoan-ny.html'

while True:
    response = requests.get(url)
    soup = bs(response.content, "lxml")

    symbols = soup.find_all('td', class_='STOCK_CODE' )
    for s in symbols:
        symbol = s.find('a').text
        print(symbol)

    next_page = soup.select_one('span', id = 'next')
    if next_page:
        next_url = next_page.get('href')
        url = urljoin(url, next_url)
    else:
        break

    print(url)
web-scraping beautifulsoup python
1 Answers

The information you want for the other pages is being returned via another call. You need to recreate that call (use your browser's network tools to see what is happening).

The request requires a token that is returned when the homepage is requested. This needs to be provided when requesting the other pages.

For example:

from bs4 import BeautifulSoup as bs
import requests

session = requests.Session()
req_homepage = session.get('https://hnx.vn/en-gb/cophieu-etfs/chung-khoan-ny.html')
soup_homepage = bs(req_homepage.content, "lxml")

for meta in soup_homepage.find_all('meta'):
    if meta.get('name', None) == '__RequestVerificationToken':
        token = meta['content']

data = {
    "p_issearch" : 0,
    "p_keysearch" : "",
    "p_market_code" : "",
    "p_orderby" : "STOCK_CODE",
    "p_ordertype" : "ASC",
    "p_currentpage" : 2,
    "p_record_on_page" : 10,
}

headers = {
    "Referer" : "https://hnx.vn/en-gb/cophieu-etfs/chung-khoan-ny.html",
    "__RequestVerificationToken" : token,
    "X-Requested-With" : "XMLHttpRequest",
}

for page in range(1, 4):
    print(f"Page {page}")
    data['p_currentpage'] = page
    req = session.post('https://hnx.vn/ModuleIssuer/List/ListSearch_Datas', data=data, headers=headers)
    json_content = req.json()['Content']
    soup = bs(json_content, "lxml")

    for td in soup.find_all('td', class_='STOCK_CODE'):
        symbol = td.find('a').text
        print(' ', symbol)

This would give you the following output:

Page 1
  AAV
  ACM
  ADC
  ALT
  AMC
  AME
  AMV
  API
  APP
  APS
Page 2
  ARM
  ART
  ATS
  BAB
  BAX
  BBS
  BCC
  BCF
  BDB
  BED
Page 3
  BII
  BKC
  BLF
  BNA
  BPC
  BSC
  BST
  BTS
  BTW
  BVS
user_49857330
Related