How access div within gridview with Beautiful Soup

user_14755028 0
Viewed 46

I'm trying to scrape all information from archdaily over multiple pages of the website (e.g. from page 1 to 20).

The html structure looks like:

    <div>
      <div class = 'afd-container-main afd-container-main--margin-bottom nft-container-main-search clearfix afd-mobile-margin search-container'>
        ::before
          <div>
            <div class='gridview'>
              <div>
                <div data-insights-category>
                  <a href = '...'> # this is the htmls i wanted

The code I'm using is

soup = BeautifulSoup(html, 'html')
for foo in soup.find_all('div'):
    bar = foo.find('div', attrs={'class': 'afd-container-main afd-container-main--margin-bottom nft-container-main-search clearfix afd-mobile-margin search-container'})
    print(bar.text)

Error message

AttributeError: 'NoneType' object has no attribute 'text'

Am I misunderstanding something?

beautifulsoup html
1 Answers

Note: Because the question does not reveal, how you get your html, it is not that easy to answer.

If you use requests, you wont get the results that way, cause the site deals with dynamic served content.

Alternativ approaches:

  • Get information with requests via api (provides even more information - categories, company,...)

    #iterate over pages
for p in range(1,3):
    r = requests.get(f'https://www.archdaily.com/search/api/v1/us/projects/categories/residential-architecture?page={p}') #url of next page
    for item in r.json()['results']:
        # iterate over results and print title+url
        print(item['title'], item['url'])

  • Get renderd html via Selenium

Example

import requests

for p in range(1,2):
    r = requests.get(f'https://www.archdaily.com/search/api/v1/us/projects/categories/residential-architecture?page={p}') #url of next page
    for item in r.json()['results']:
        print(item['title'], item['url'])

Output

Wooden House / derksen | windt architecten https://www.archdaily.com/972995/wooden-house-derksen-windt-architecten?ad_source=search&ad_medium=projects_tab
PLA2 House / Dersyn Studio https://www.archdaily.com/972939/pla2-house-dersyn-studio?ad_source=search&ad_medium=projects_tab
gjG House / BLAF Architecten https://www.archdaily.com/951845/gjg-house-blaf-architecten?ad_source=search&ad_medium=projects_tab
Leopoldo 1201 Residential Building / aflalo/gasperini arquitetos https://www.archdaily.com/972959/leopoldo-1201-residential-building-aflalo-gasperini-arquitetos?ad_source=search&ad_medium=projects_tab
Sayang House / Carlos Gris Studio https://www.archdaily.com/972773/sayang-house-carlos-gris-studio?ad_source=search&ad_medium=projects_tab
Nong Ho 17 House / Skarn Chaiyawat https://www.archdaily.com/972911/nong-ho-17-house-skarn-chaiyawat?ad_source=search&ad_medium=projects_tab
LÂM’s Home / AD+studio https://www.archdaily.com/972794/lams-home-ad-plus-studio?ad_source=search&ad_medium=projects_tab
Limestone House / John Wardle Architects  https://www.archdaily.com/972958/limestone-house-john-wardle-architects?ad_source=search&ad_medium=projects_tab
Quay Wall House / Thomas Kemme Architects https://www.archdaily.com/971781/quay-wall-house-thomas-kemme-architects?ad_source=search&ad_medium=projects_tab
...
user_144608240
Related