How would I scrape the sic code description?

user_8684461 0
Viewed 332

Hi I am using BS4 to scrape the sic codes and descriptions. I currently have the following code which does exactly what I want but I don't know how to scrape the description pictures below in the inspect element view as well as the view source.

To be clear the bit I want is "State commercial banks" and "LABORATORY ANALYTICAL INSTRUMENTS"

https://www.sec.gov/cgi-bin/browse-edgar?CIK=866054&owner=exclude&action=getcompany&Find=Search

<div class="companyInfo">
     <span class="companyName">COMMERCIAL NATIONAL FINANCIAL CORP /PA <acronym title="Central Index Key">CIK</acronym>#: <a href="/cgi-bin/browse-edgar?action=getcompany&amp;CIK=0000866054&amp;owner=exclude&amp;count=40">0000866054 (see all company filings)</a></span>
     <p class="identInfo"><acronym title="Standard Industrial Code">SIC</acronym>: <a href="/cgi-bin/browse-edgar?action=getcompany&amp;SIC=6022&amp;owner=exclude&amp;count=40">6022</a> - STATE COMMERCIAL BANKS<br />State location: <a href="/cgi-bin/browse-edgar?action=getcompany&amp;State=PA&amp;owner=exclude&amp;count=40">PA</a> | State of Inc.: <strong>PA</strong> | Fiscal Year End: 1231<br />(Office of Finance)<br />Get <a href="/cgi-bin/own-disp?action=getissuer&amp;CIK=0000866054"><b>insider transactions</b></a> for this <b>issuer</b>.


for cik_num in cik_num_list:
try:
    url = r"https://www.sec.gov/cgi-bin/browse-edgar?CIK={}&owner=exclude&action=getcompany".format(cik_num)
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    try:
        comp_name = soup.find_all('div', {'class':'companyInfo'})[0].find('span').text
        sic_code = soup.find_all('p', {'class':'identInfo'})[0].find('a').text

enter image description here enter image description here

web-scraping beautifulsoup python
1 Answers
import requests
from bs4 import BeautifulSoup

url = 'https://www.sec.gov/cgi-bin/browse-edgar?CIK=866054&owner=exclude&action=getcompany&Find=Search'

soup = BeautifulSoup(requests.get(url).content, 'html.parser')

sic_code_desc = soup.select_one('.identInfo').a.find_next_sibling(text=True).split(maxsplit=1)[-1]
print(sic_code_desc)

Prints:

STATE COMMERCIAL BANKS

For url = 'https://www.sec.gov/cgi-bin/browse-edgar?CIK=1090872&owner=exclude&action=getcompany&Find=Search' it prints:

LABORATORY ANALYTICAL INSTRUMENTS
user_100359850
Related