I wrote code for a webscraping data. The idea is to scrape links and then get into each and get the same data for x elements.
So I wrote the code but:
- Couldn't get only text from td classes (dont have much html knowledge). I get as a result: the name of td class and the middle"93203587600000" and want only the middle. The .text method doesnt work as it is soup.find_all doesnt have it. How to do that?
- I want to export it to the excel spreadshit so I can investigate it but a table I get consists of same elements. So idea is to name each with a name (example : name1, name2, name3) and then put different data into them in the same order. How to do that and then export to Excel?
MY CODE:
import requests
from bs4 import BeautifulSoup
import pandas as pd
baseURL = ""
firmy = []
for x in range(2, 3):
r = requests.get(f"http://={x}") //comment: frame for looping through sites
soup = BeautifulSoup(r.content, features="html.parser")
nazwy = soup.find_all('div', class_='col-9')
for item in nazwy:
for link in item.find_all('a', href=True):
firmy.append(link['href'])
# testlink = ""
daneFirm = []
for link in firmy:
r = requests.get(link)
soup = BeautifulSoup(r.content, features="html.parser")
basicData = soup.find('div', class_='basicData').text.strip()
daneFirm.append(basicData.text)
print(daneFirm)
What I get as a result is below but I want it as described. Any hints?
[[<td class="col-xs-8">xxx</td>, <td class="col-xs-8"></td>, <td class="col-xs-8">xxx</td>, <td class="col-xs-8"></td>, <td class="col-xs-8">xxxx</td>, <td class="col-xs-8" itemscope="" itemtype="http://data-vocabulary.org/Address">
<span itemprop="street-address">KARMELKOWA 29/41 </span><br/>
53-437 <span itemprop="locality">WROCŁAW</span><br/>
The data I need are in below html code:
<divclass ="basicData">
<table id="basicDataTable">
<tbody>
<tr>
<th>Status</th
<td>Aktywna</>
</tr>
<tr>
<th class="col-xs-4">NIP</th>
<td class="col-xs-8">5220003782</td>
and about 10 x the same element. And what I need is basically in the "col-xs-8" or table body's.
