How to find_all(id) from a div with beautiful soup in python

user_8257514 0
Viewed 8943

I want to print out all the IDs from a page which has a unique class.

The page what I want to scrape with Beautiful Soup is like this: 

<div itemscope itemprop="item" itemtype="http://schema.org/Product" id="12345" class="realestate"> 
<div class="contentArea"> 
<meta itemprop="name" content="Name - 12345 " /> 
<meta itemprop="url" content="https://url12345.hu" />   
<meta itemprop="category" content="category1" />   
</div>
</div>
<div itemscope itemprop="item" itemtype="http://schema.org/Product" id="12346" class="realestate"> 
<div class="contentArea"> 
<meta itemprop="name" content="Name - 12346 " /> 
<meta itemprop="url" content="https://url12346.hu" />   
<meta itemprop="category" content="category1" />   
</div>
</div>

the 'ID' is a unique identifier from the Itemscope DIVs, so somehow I want to extract these uniques IDs and print all of them out (the reson is to attach all other ad information to this ID (like name, URL, etc.) later)

I tried with this python code, but it does not work. 

import requests
from bs4 import BeautifulSoup

page = requests.get('searchResultPage.url')
soup = BeautifulSoup(page.text, 'html.parser')
id = soup.find_all('id')
print(id)

It gives back an empty list.

What I expect, and what I want is to get back a list with the ID-s from the divs, this way: 12345 12346

Thanks for your help in advance!

beautifulsoup python
5 Answers

HS-nebula is correct that find_all looks for tags of a certain type, in your soup id is an attribute not a type of tag. To get a list of all id's in the soup you can use the following one liner 

ids = [tag['id'] for tag in soup.select('div[id]')]

this uses CSS selectors instead of bs4's find_all since I find bs4's docs regarding its built-ins lacking.

So what the soup.select does is return a list of all div elements which have an attribute called 'id' then we loop over that list of div tags and add the value of the 'id' attribute to the ids list.

user_103598500

If you wanted to see all IDs in the whole web URL this will work but it will also include lots of the outer and inner HTML tags and code.

id = soup.find_all(id=True)
print(id)

If you want to see the actual IDs without all the HTML in a list/array of one ID per a row here is an option:

for ID in soup.find_all('div', id=True):  
    print(ID.get('id'))

In the above For Loop, you are specifying the tag in quote marks, i.e. 'div' and you then are asking it to list the attribute you want, i.e. 'id=True'

user_158258390

BeautifulSoup's find_all() function finds all HTML tags of a certain kind. id is not a tag, it's an attribute of a tag. You have to search for the tags that contain the IDs that you want, in this case, the div tag.

div_tags = soup.find_all('div')
ids = []
for div in div_tags:
     ID = div.get('id')
     if ID is not None:
         ids.append(ID)

BeautifulSoup also provides the capability to find tags with specific attributes.

user_108528410

There is a difference between tag and attribute, in your case div is the tag and id is the attribute of the tag. So you must use find_all(name='tag') to find all the tags and after you can use get('attribute') to get the attribute. If you want to scrape long pages, you can optimize a bit your code by using comprehension list:

soup = BeautifulSoup(markup=page, 'html.parser')
test = [r['id'] for r in soup.find_all(name="div", attrs={"id":"12346"}) if r.get('id') is not None]

Output:

['12345', '12346']

Moreover, you can use find_all() to get all tags that do have an id attribute (thank you Jon Clements), for example:

test = [r['id'] for r in soup.find_all(name="div", attrs={"id":True})]
user_117242280

Here are some solutions : If only tags with id to be considered:

tags = page_soup.find_all(id=True)
for tag in tags:
    print(tag.name,tag['id'],sep='->')

If need to loop all tags:

 tags = page_soup.find_all()
    for tag in tags:
        if 'id' in tag.attrs:
            print(tag.name,tag['id'],sep='->')

Just to get all ids only:

ids =[tag['id'] for tag in page_soup.find_all(id=True)]
user_2326870
Related