How to obtain link inside of a DIV nested in a TD with BeautifulSoup

user_16051077 0
Viewed 309

Problem: "Easy" way (pd.read_html()) of obtaining table information with Pandas isn't working for my use case.

It's only pulling what I believe is the label text, and it's got this newb confuzzled. What I need is at least the link (to pdf) text.

Table was obtained from an ASPX page via Requests/BeautifulSoup. I was able to get that table into a Pandas DataFrame without issue.

If you use the link below, please copy and paste it as to remove the referrer URL. My luck, some IT guy changes the code, breaking the script earlier than needed. lol

Link to page (you'll have to manually search using the variables defined in the script).

Scraper.py:

import requests
import pandas as pd
from lxml import html
from bs4 import BeautifulSoup as bs

# User-defined variables
SearchBy = 'DateFiled'
FiledStartDate = '2020-01-01'
FiledEndDate = '2020-01-01'
County = 'Luzerne'
MDJSCourtOffice = 'MDJ-11-1-01'

host = "ujsportal.pacourts.us"
base_url = "https://" + host
search_url = base_url + "/CaseSearch"

# Headers are required. Do not change.
headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,\
               image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Connection': 'keep-alive',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Host': host,
    'Origin': base_url,
    'Referer': search_url,
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) \
                   Gecko/20100101 Firefox/88.0'
}

# Open session request to obtain proper cookies
ses = requests.session()
req = ses.get(search_url, headers=headers)

# Get required hidden token so we can search
tree = html.fromstring(req.content)
veri_token = tree.xpath("/html/body/div[3]/div[2]/div/form/input/@value")[0]

# Import search criteria from user-defined variables
payload = {
    'SearchBy': SearchBy,
    'AdvanceSearch': 'true',
    'FiledStartDate': FiledStartDate,
    'FiledEndDate': FiledEndDate,
    'County': County,
    'MDJSCourtOffice': MDJSCourtOffice,
    '__RequestVerificationToken': veri_token
}

# Make search request
results = ses.post(
    search_url,
    data=payload,
    headers=headers
)

# Save html page to disk
with open("tmp/test_draft1.html", "w") as f:
    f.write(results.text)

# Open local HTML page for processing
with open("tmp/test_draft1.html") as html:
    page = bs(html, 'lxml')

table = page.find('table', {'id': 'caseSearchResultGrid'})

# Save table as sperate HTML for later audit
with open("tmp/test_draft1_table.html", "w") as f:
    f.write(table.prettify())


# Remove unneeded tags so we don't have to do it in Pandas
def clean_tags(page):
    for tag in table.select('div.bubble-text'):
        tag.decompose()
    for tag in table.select('div.modal'):
        tag.decompose()
    for tag in table.find_all(['th', 'tr', 'td'], class_="display-none"):
        tag.decompose()
    for tag in table.select('tfoot'):
        tag.decompose()


clean_tags(page)

# Start constructing dataset
columns = table.find('thead').find_all('th')
column_names = [c.get_text() for c in columns]

table_rows = table.find('tbody').find_all('tr')

case_info = []
for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.get_text() for tr in td]
    case_info.append(row)

# Forward dataset to Pandas for analysis
df = pd.DataFrame(case_info, columns=column_names)
df.columns.values[16] = "Docket URL"

if SearchBy == 'DateFiled':
    df.drop(columns=['Event Type',
            'Event Status', 'Event Date', 'Event Location'], inplace=True)

df
exit("Scrape Complete!")

This works to pull the docket pdf link by itself into a seperate list. But does not update the cell properly.

for row in table_rows:
    row_processed = []
    cells = row.find_all("td")
    if len(cells) == 17:
        docket_url = base_url + cells[16].find('a')['href']
        row_processed.append(docket_url)

Current print(df) truncated output snippet:

              Docket Number  ...                 Docket URL
0  MJ-11101-CR-0000001-2020  ...  Docket SheetCourt Summary
1  MJ-11101-CR-0000003-2020  ...  Docket SheetCourt Summary
2  MJ-11101-CR-0000006-2020  ...  Docket SheetCourt Summary
3  MJ-11101-NT-0000081-2020  ...  Docket SheetCourt Summary

Needed print(df) truncated output snippet:

              Docket Number  ...                 Docket URL
0  MJ-11101-CR-0000001-2020  ...  https://link/to/docketPDF
1  MJ-11101-CR-0000003-2020  ...  https://link/to/docketPDF
2  MJ-11101-CR-0000006-2020  ...  https://link/to/docketPDF
3  MJ-11101-NT-0000081-2020  ...  https://link/to/docketPDF
pandas web-scraping beautifulsoup python
1 Answers

Ok, with the help and guidance of QHarr in the comments of the original question, I was able to come up with a solution. As with anything coding related, I am sure this isn't the only answer.

Anyway... After trying to get these two iteration loops integrated the way I wanted, concatenating two Pandas DataFrames works.

Case Info Data:

case_info = []
for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.get_text() for tr in td]
    case_info.append(row)

Docket URLs: (produces a list with the urls)

docket_urls = []
for drow in table_rows:
    docket_sheets = []
    cells = drow.find_all("td")
    if len(cells) == 17:
        docket_url = base_url + cells[16].find('a')['href']
        docket_sheets.append(docket_url)
    docket_urls.append(docket_sheets)

DataFrames:

# Import case info dataset to Pandas
df_case_info = pd.DataFrame(case_info, columns=column_names)
df_case_info.columns.values[16] = "Docket Text"  # Rename col = easy to drop

df_case_info.drop(columns=['Docket Text'], inplace=True)

if SearchBy == 'DateFiled':
    df_case_info.drop(columns=['Event Type',
            'Event Status', 'Event Date', 'Event Location'], inplace=True)

# Import docket URLs into Pandas
df_docket_urls = pd.DataFrame(docket_urls, columns={'Docket URL'})

# Concatonate both DataFrames into one
df_mdj = pd.concat([df_case_info, df_docket_urls], axis=1)

I still want to learn how to do this the way I originally planned. But if it's not broke, don't fix it, right?

To those who helped, Thank You. This is what I call a Suckcess. :)

user_160510770
Related