I need to scrape this website:
https://sec.report/Ticker/AAPL
I need to get the CIK number 0000320193
When I do soup.prettify, it just says it needs to use javascript. Also, I don't want to open a web browser because it needs to be automated
I need to use the python beautiful soup and requests library
To get correct response from server, set correct User-Agent HTTP header:
import requests
from bs4 import BeautifulSoup
url = 'https://sec.report/Ticker/AAPL'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}
soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')
print(soup.h2.text) # or print(soup.h2.text.split()[-1]) for "0000320193"
Prints:
SEC CIK 0000320193