Webscraping bs4 - sorting resaults from different URLs into table

Question

I have written the script below to scrape a website. I have left out the URL, if you need this write to me and i will supply it to you.

The current output is kinda messy but it does the job. Im very novice at scraping so if you have any suggestions on how to improve the scraping it selfe please tell me.

Im looking for help to structure the results into a table that looks like this:

| source | columns... | | -------- | -------------- | | url1 | values | | url2 | values | Columns: Antal aktier, Börsvärde MSEK, Direktavkastning %, P/E-tal, P/S-tal, etc...

values from data1: 59840000, 5084,00, 0,00,11,11, 0,59, etc...

values from data2: 14532434, 2284,50, 2,70, 9,73, 0,52, etc...

Ides on how to solve this is very welcome.

Script:

import bs4 
import requests
import re
from bs4 import BeautifulSoup as bs

URL1 = "XXX"
URL2 ="YYY"

r1 = requests.get(URL1)
r2 = requests.get(URL2)

soup1 = bs(r1.content)
soup2 = bs(r2.content)

data1 = soup1.find_all('dl', attrs= {"class": "border XSText rightAlignText noMarginTop highlightOnHover thickBorderBottom noTopBorder"})
data2 = soup2.find_all('dl', attrs= {"class": "border XSText rightAlignText noMarginTop highlightOnHover thickBorderBottom noTopBorder"})


print(data1[1])
print(data2[1])

Webscraping output:

Antal aktier
59 840 000
Börsvärde MSEK
5 084,00
Direktavkastning %
0,00
P/E-tal
11,11
P/S-tal
0,59
Kurs/eget kapital 
2,60
Omsättning/aktie SEK
132,00
Vinst/aktie SEK
6,98
Eget kapital/aktie SEK
29,55
Försäljning/aktie SEK
-
Effektivavkastning %
0,00
Antal ägare hos Avanza
16 041


Antal aktier
14 532 434
Börsvärde MSEK
2 284,50
Direktavkastning %
2,70
P/E-tal
9,73
P/S-tal
0,52
Kurs/eget kapital 
2,73
Omsättning/aktie SEK
303,47
Vinst/aktie SEK
16,16
Eget kapital/aktie SEK
58,34
Försäljning/aktie SEK
-
Effektivavkastning %
2,70
Antal ägare hos Avanza
3 994
```

user_4582696 · Answer

There are many ways you can solve this! Looking at your output tells us that

elements hold the column names and

elements hold the values. So we can iterate through them and append the data to lists.

column_list = []
value_list = []

columns = soup1.find_all('dt')
for col in columns:
    column_list.append(col.text.strip()) # strip() removes extra space from the text    

values = soup1.find_all('dd')
for val in values:
    value_list.append(val.text.strip())

for i in range(len(column_list)):
    print(column_list[i] + ': ' + value_list[i])

Now you can use the data in your lists as you wish. It currently gives an output likes this:

Kortnamn: AAPL
ISIN: US0378331005
Marknad: NASDAQ
Bransch: Teknik
Handlas i: USD
Beta: 1,1927
Volatilitet %: 24,99
Belåningsvärde %: 60
Säkerhetskrav %: 150
Superränta: Ja
Blankningsbar: Nej
Antal aktier: 17 001 802 000
Börsvärde MUSD: 2 226 555,99
Direktavkastning %: 0,62
P/E-tal: 38,24
P/S-tal: 8,16
Kurs/eget kapital: 31,05
Omsättning/aktie USD: 16,05
Vinst/aktie USD: 3,42
Eget kapital/aktie USD: 4,25
Försäljning/aktie USD: -
Effektivavkastning %: 0,62
Antal ägare hos Avanza: 34 331

Webscraping bs4 - sorting resaults from different URLs into table

1 Answers