I'm unable to parse past the div id= "id="divTradeHaltResults". When I try to return the table within this div I get None. Thanks in advance!
from bs4 import BeautifulSoup
import requests
my_url = "https://www.nasdaqtrader.com/Trader.aspx?id=TradeHalts"
r = requests.get(url=my_url)
page_text = r.text
soup = BeautifulSoup(page_text, "lxml")
table = soup.table
print(table)
If you select that tag inside the soup, you get the tag but it's empty. If you look on the webpage, you can see the table in the tag. My guess is that this table is generated with JS (in some form), thus it does not come with the HTML. My solution would be to turn to something like Selenium.
This is the code I ran to select that tag:
soup.find('div', {'id':'divTradeHaltResults'})
# <div id="divTradeHaltResults"></div>
If you look into the JS on the page, you can actually find the function that generates the table, as I mentioned above:
function GetTradeHalts()
{
document.getElementById('divTradeHaltResults').innerHTML = "updating....";
Server.BL_TradeHalt.GetTradeHalts(cb_GetTradeHalts);
setTimeout(GetTradeHalts, 1000 * 60);
}