Scraping table with href links with unstructured data

user_14164479 0
Viewed 48

I am trying to extract some information from a table which appears on a webpage. I am able to extract all the data as wanted but get stuck at scraping the pdf links inside.

This is my source code:

Here is my code working so far:

    import pandas as pd
    import requests
    from bs4 import BeautifulSoup
    
    hdr={'User-Agent':'Chrome/84.0.4147.135'}
    
    dfs=[]
    
    for page_number in range(0,10):
        http= "https://example.com/portal/ap_list_all.php?sPage={}".format(page_number)
    
        print('Downloading page %s...' % http)
        url= requests.get(http,headers=hdr)
        soup = BeautifulSoup(url.text, 'html.parser')
        table = soup.find('table')
        df_list= pd.read_html(url.text)
        df = pd.concat(df_list)
        df['Link'] = [link.get('href') for link in table.find_all('a')]
        dfs.append(df)
    
    final_df = pd.concat(dfs)
    final_df.to_csv('myscraper.csv',index=False,encoding='utf-8-sig')

However, this is what I get: enter image description here

The result I want is as below, coupled with the additional column of pdf links:

enter image description here

Any help would be appreciated. Thank you so much in advance.

pandas web-scraping dataframe python
1 Answers

You can use this example how to scrape the data into panda's dataframe and save it to CSV file:

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://ejudgment.kehakiman.gov.my/portal/ap_list_all.php?sPage={page}&sRecordsPerPage=20&sSortField=aph_ap_date&sSortOrder=DESC"

data = []
for page in range(0, 10):
    print("Getting page {}..".format(page))

    soup = BeautifulSoup(
        requests.get(url.format(page=page)).content, "html.parser"
    )

    headers = [
        h.get_text(strip=True)
        for h in soup.select("thead tr#trchk2 td, thead tr#trchk2 th")
    ]

    for row in soup.select('tbody tr[id^="trchk"]'):
        dic = {}
        for h, d in zip(headers, row.select("td")):
            dic[h] = d.get_text(strip=True, separator="\n")
        dic["Link"] = "https:" + row.find("a", target="_blank")["href"]
        data.append(dic)

df = pd.DataFrame(data)
print(df)
df.to_csv("data.csv", index=False)

Prints:

        #                                         Nombor Kes                                              Pihak                                         Kata Kunci Tarikh Alasan Penghakiman                                  Hakim / Majistret                                            Dokumen                                               Link
0      1.  WA-42(R)(A)-4-11/2018; WA-42R-13-11/2018; WA-4...  PERAYU  : \r\n\t\t1. MOHD ARIF BIN AB RAHMAN\n...  Appellants found guilty of corruption in Sessi...                28/04/2021                           YA DATUK ASLAM ZAINUDDIN  Alasan Penghakiman\n1)\n28042021 judgment case...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
1      1.                                   JA-22C-3-02/2020  PLAINTIF  : \r\n\t\tYPJ PLANTATIONS SDN BHD\n(...  Civil procedure - Application to set aside JID...                28/04/2021                      YA PUAN EVROL MARIETTE PETERS  Alasan Penghakiman\n1)\nja-22c-3-02-2020 (ypj ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
2      1.                             W-02(C)(A)-163-01/2020  PERAYU  : \r\n\t\tAHMAD ZAKI SDN BHD\n(COMPANY...                                         Ahmad Zaki                27/04/2021  YA DATO' ABU BAKAR BIN JAIS\nKorum:\nYA DATO' ...     Alasan Penghakiman\n1)\nahmad zaki sdn bhd.pdf  https://ejudgment.kehakiman.gov.my/ks_builtin/...
3      1.                                  WA-45B-24-01/2018  PENDAKWARAYA  : \r\n\t\tPN. ZAHIDA BINTI ZAKAR...                                             GROUND                26/04/2021                   YA TUAN COLLIN LAWRENCE SEQUERAH  Alasan Penghakiman\n1)\npp v muhammad khairuan...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
4      1.                                      62-01-02/2019  PENDAKWARAYA  : \r\n\t\tPENDAKWARAYA\nTERTUDUH...  -seksyen 326 Kanun Keseksaan- cedera parah- kr...                26/04/2021                                TN MUSYIRI BIN PEET  Alasan Penghakiman\n1)\nalasan penghakiman thu...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
5      1.                                   JA-25-40-12/2020  PEMOHON  : \r\n\t\tKIND ACTION (M) SDN BHD\n(N...                Leave for judicial review - granted                25/04/2021             YA DATO' FREDRICK INDRAN X.A. NICHOLAS  Alasan Penghakiman\n1)\nmtjbs2-ja-25-40-12-202...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
6      1.                               WA-22NCC-618-12/2020  PLAINTIF  : \r\n\t\tTAN MAN HIN (NRIC NO.55082...  Summary Judgment application dismissed - wheth...                24/04/2021                              YA TUAN ONG CHEE KWAN  Alasan Penghakiman\n1)\nwa-22ncc-618-12-2020 t...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
7      1.                               WA-22NCC-167-04/2020  PLAINTIF  : \r\n\t\tSMEB ASSET MANAGEMENT SDN....                                               O.14                23/04/2021            YA TUAN WAN MUHAMMAD AMIN BIN WAN YAHYA  Alasan Penghakiman\n1)\njudgment [sgd] (smeb) ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
8      1.                               WA-22NCC-401-08/2020  PLAINTIF  : \r\n\t\tKHEE SAN BERHAD &AMP 1 LAG...                                                  -                22/04/2021                         YA PUAN LIZA CHAN SOW KENG  Alasan Penghakiman\n1)\nkhee san dan 1 lagi v ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
9      1.                                  BA-12B-24-02/2020  PERAYU  : \r\n\t\tLETCHUMY A/P KRISHNAN\nRESPO...                          Letchumy v. Khairil Azraf                22/04/2021                                   TN TEE GEOK HOCK  Alasan Penghakiman\n1)\ngoj - [final] khairil ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
10     1.                                  BA-12B-92-06/2019  PERAYU  : \r\n\t\tKHAIRIL AZRAF BIN MOHD SEDEK...                           Khairil Azraf v Letchumy                22/04/2021                                   TN TEE GEOK HOCK  Alasan Penghakiman\n1)\ngoj - [final] khairil ...  https://ejudgment.kehakiman.gov.my/ks_builtin/...
11     1.                               JA-41(ORS)-2-11/2020  PERAYU  : \r\n\t\tMUHAMMAD BIN JAAFAR\nRESPOND...                 Seksyen 413 Kanun Tatacara Jenayah                22/04/2021                      YA DR. SHAHNAZ BINTI SULAIMAN    Alasan Penghakiman\n1)\nmuhammad bin jaafar.pdf  https://ejudgment.kehakiman.gov.my/ks_builtin/...

...

And saves data.csv (screenshot from LibreOffice):

enter image description here

user_100359850
Related