HTML parts locating

user_18735127 0
Viewed 109

I am trying to extract each row individually to eventually create a dataframe to export them into a csv. I can't locate the individual parts of the html.

I can find and save the entire content (although I can only seem to save this on a loop so the pages appear hundreds of times), but I can't find any html parts nested beneath this. My code is as follows, trying to find the first row:

    
    page = requests.get(url)
    soup = BeautifulSoup(page.text, 'html.parser')

    content = soup.find('div', {'class': 'view-content'})

    for infos in content:
        try:
            data = infos.find('div', {'class': 'type type_18'}).text
        except:
            print("None found")

df = pd.DataFrame(data)

df.columns = df.columns.str.lower().str.replace(': ','')

df[['type','rrr']] = df['rrr'].str.split("|",expand=True)

df.to_csv (r'savehere.csv', index = False, header = True)

This code just prints "None found" because, I assume, it hasn't found anything else to print. I don't know if I am not finding the right html part or what. Any help would be much appreciated.

pandas web-scraping dataframe beautifulsoup python
2 Answers
What happens?

Main issue here is that content = soup.find('div', {'class': 'view-content'}) is no ResultSet and contains only a single element. Thats why your second loop only iterates once.

Also Caused by this behavior you will swap from beautifoulsoup method find() to python string method find() and these two are operating in a different way - Without try/except you will see the what is going on, it try to find a string:

for x in soup.find('div', {'class': 'view-content'}):
    print(x.find('div'))

Output

...
-1
<div class="views-field views-field-title-1"> <span class="views-label views-label-title-1">RRR: </span> <span class="field-content"><div class="type type_18">Eleemosynary grant</div>2256</span> </div>
...
How to fix?

Select your elements more specific in this case the views-row:

sections = soup.find_all('div', {'class': 'views-row'})

While you iterate each section you could select expected value:

sections = soup.find_all('div', {'class': 'views-row'})

for section in sections:
    print(section.select_one('div[class*="type_"]').text)
Example

Is scraping all the information and creates DataFrame

import requests
from bs4 import BeautifulSoup
import pandas as pd

data = []
website = #link here#
    

    page = requests.get(url)
    soup = BeautifulSoup(page.text, 'html.parser')

    sections = soup.find_all('div', {'class': 'views-row'})

    for section in sections:
        d = {}
        for row in section.select('div.views-field'):
            d[row.span.text] = row.select_one('span:nth-of-type(2)').get_text('|',strip=True)
        data.append(d)

df = pd.DataFrame(data)

### replacing : in header and set all to lower case
df.columns = df.columns.str.lower().str.replace(': ','')

...

user_144608240

I think that You wanted to make pagination using for loop and range method and to grab RRR value.I've done the next pages meaning pagination in long url.

from bs4 import BeautifulSoup
import requests
import pandas as pd

url = #insert url#
data=[]
for page in range(1,7):
    req=requests.get(url.format(page=page))

    soup = BeautifulSoup(req.content,'lxml')
    
    for r in soup.select('[class="views-field views-field-title-1"] span:nth-child(2)'):

        rr=list(r.stripped_strings)[-1]
        #print(rr)
        
        data.append(rr)


df = pd.DataFrame(data,columns=['RRR'])
print(df)
#df.to_csv('data.csv',index=False)

Output:

List

user_128484110
Related