How to extract table value from using BeautifulSoup

user_13770157 0
Viewed 51

I am trying to extract the Solar Longitude value from this table

I am using this code to look at the structure of the table:

import requests
from bs4 import BeautifulSoup
URL = 'https://viewer.mars.asu.edu/viewer/themis#P=V77388006&T=2'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.prettify())

However, when I look at the output and try to find the Solar Longitude it is not there. I even tried to saved the output of the code as a .txt file and got the same result. I did notice that my output is a lot shorter than the actual HTML code I see in the browser.

Am I missing something?

web-scraping beautifulsoup python-3.x
1 Answers

You may get not all content back with requests, cause it is served dynamically by the website, but you can use selenium to fix that.

Example

from selenium import webdriver
from bs4 import BeautifulSoup


driver = webdriver.Chrome(executable_path=r'C:\Program Files\ChromeDriver\chromedriver.exe')
url = 'https://viewer.mars.asu.edu/viewer/themis#P=V77388006&T=2'
driver.get(url)

soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.close()

soup.select_one('[data-field="Solar Longitude"]').parent.nextSibling.get_text()

Output

30.528906°
user_144608240
Related