Scraping a dynamically loaded table with BeautifulSoup

user_11181986 0
Viewed 89

My code could return values of first two tags, but the behind won't in per tag.

HTML: enter image description here

My code: import bs4 as bs import requests

resp = requests.get('https://q.stock.sohu.com/cn/bk_4401.shtml')
resp.encoding = 'gb2312'
soup = bs.BeautifulSoup(resp.text, 'lxml')
tab_sgtsc_list = soup.find('table').find('tbody').find_all('tr')

for tab_sgtsc in tab_sgtsc_list:
    print('**************************************')
    print(tab_sgtsc.find_all('td')[0].text)
    print(tab_sgtsc.find_all('td')[1].text)
    print(tab_sgtsc.find_all('td')[2].text)
    print(tab_sgtsc.find_all('td')[3].text)
    print('**************************************')

Result: enter image description here

python-requests beautifulsoup python
1 Answers

The table is rendered dynamically by JavaScript so you won't get much from pure HTML.

However, selenium and pandas come to the rescue!

Required:

Here's how:

import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

driver.get("https://q.stock.sohu.com/cn/bk_4401.shtml")

wait = WebDriverWait(driver, 10)
element = wait.until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, 'table.tableMSB'))
).text.replace("点击按代码排序查询", "").split()

table = [element[i:i + 12] for i in range(0, len(element), 12)]
pd.DataFrame(table[1:], columns=table[0]).to_csv("your_table_data.csv", index=False)

Output:

enter image description here

user_61067910
Related