Python web text list to dataframe

user_14772220 0
Viewed 122

I have several values scraped from a website, and I wanna covert to dataframe like:

dates          titles     links
2021-05-13      AAA       https

I use:

html_link = 'https://www.ksei.co.id/publications/new-securities-registration?setLocale=en-US'
html = requests.get(html_link).text
soup = BeautifulSoup(html, 'html.parser')
titles = []  
links = []
dates = []
for ultag in soup.find_all('ul', {'class': 'list-nostyle'}):
    for litag in ultag.find_all('li'):
        for dates in litag.find_all('small', {'class': 'muted'}):
                title = litag.find('h2', {'class': 'h4 no-margin'})
                link = litag.find('a', href=True)
                titles.append(title.text)
                links.append(f"https://www.pds.com.ph{link['href']}")
                dates.append(dates.text)
                #print(titles,links,dates)
dataframe = pd.DataFrame(zip(titles, links, dates), columns=['Titles', 'Links', 'Dates'])
print(dataframe)

But it only returns the first two rows, no idea why. I am confused with zip function, and always stuck with the lists to dataframe...

Thanks for the help in advance!

python
3 Answers

You're making this way too complicated than it should be. And by that I mean those nested for loops.

Just grab all the "boxes" and scoop out all the parts you need. Finally, dump the list of lists to a DataFrame and you're all done!

Here's how:

import pandas as pd
import requests
from bs4 import BeautifulSoup

html_link = 'https://www.ksei.co.id/publications/new-securities-registration?setLocale=en-US'
boxes = BeautifulSoup(requests.get(html_link).text, 'html.parser').select(".box--medium")

data = []
for box in boxes:
    title = box.find("h2").getText()
    date = box.find("b").getText().replace(", ", " ")
    name = box.find("p").getText()
    link = f'https://www.pds.com.ph{box.find("a")["href"]}'
    data.append([title, date, name, link])

df = pd.DataFrame(data, columns=['Titles', 'Dates', 'Names', 'Links'])
print(df.head())
df.to_csv("your_data.csv", index=False)

Output:

                Titles  ...                                              Links
0  KSEI-3512/DIR/0521   ...  https://www.pds.com.ph/Announcement/Files/1271...
1  KSEI-3482/DIR/0521   ...  https://www.pds.com.ph/Announcement/Files/1270...
2   KSEI-7362/JKU/0521  ...  https://www.pds.com.ph/Announcement/Files/1270...
3  KSEI-3440/DIR/0521   ...  https://www.pds.com.ph/Announcement/Files/1269...
4   KSEI-3394/DIR/0521  ...  https://www.pds.com.ph/Announcement/Files/1268...

[5 rows x 4 columns]

and a .csv file:

enter image description here

user_61067910

The problem is because you have assigned same variable name(dates) to two data structures.there is a list assigned with name dates and also in the third loop you've dates variable.so, zip function is taking dates variable from the for loop

html_link = 'https://www.ksei.co.id/publications/new-securities-registration?setLocale=en-US'
html = requests.get(html_link).text
soup = BeautifulSoup(html, 'html.parser')
titles = []
links = []
dates_1 = []
for ultag in soup.find_all('ul', {'class': 'list-nostyle'}):
    for litag in ultag.find_all('li'):
        for dates in litag.find_all('small', {'class': 'muted'}):
            title = litag.find('h2', {'class': 'h4 no-margin'})
            link = litag.find('a', href=True)
            titles.append(title.text)
            links.append(f"https://www.pds.com.ph{link['href']}")
            print(dates.text)
            print(dates)
            dates_1.append(dates.text)

dataframe = pd.DataFrame(zip(titles, links, dates_1), columns=['Titles', 'Links', 'Dates'])
user_81722170

Variables are reused and zip will take the shortest

import itertools


html_link = 'https://www.ksei.co.id/publications/new-securities-registration?setLocale=en-US'
html = requests.get(html_link).text
soup = BeautifulSoup(html, 'html.parser')
titles = []  
links = []
dates = []
for ultag in soup.find_all('ul', {'class': 'list-nostyle'}):
    for litag in ultag.find_all('li'):
        for dat in litag.find_all('small', {'class': 'muted'}):
                title = litag.find('h2', {'class': 'h4 no-margin'})
                link = litag.find('a', href=True)
                titles.append(title.text)
                links.append(f"https://www.pds.com.ph{link['href']}")
                dates.append(dat.text)
                #print(titles,links,dates)
dataframe = pd.DataFrame(itertools.zip_longest(titles, links, dates), columns=['Titles', 'Links', 'Dates'])
print(dataframe)
user_115079100
Related