bs4 search a word in all attributes of a given tag

user_14017906 0
Viewed 144

I was developing a web scraper to find the price tag from a given website.

I have a code

price = soup.findAll(['div'],{'class':re.compile(r'(.*?price.*?)',re.IGNORECASE)})

With this, I was able to find all div tags with a class attribute having values with a price keyword. (words which contain price like - data-price, price value etc.)

But I want to retrieve all div tags which contain price keyword irrespective of the attribute name.

Example:

The Website I am trying to scrape has this format:

<div class="css-2vqe5n esdkp3p0" data-automation="buybox-price" aria-label="Now $74">$74</div>

My code only retrieves if the price keyword is present in the class attribute, But in this case, it is present in the data-automation attribute.

So I am looking for a solution that would search all attributes of the div tag but not only in the class tag.

selenium-webdriver web-scraping beautifulsoup html python
1 Answers

For this task you can use .find_all() with custom function.

For example:

from bs4 import BeautifulSoup


html_text = '''
<div class="css-2vqe5n esdkp3p0" data-automation="buybox-price" aria-label="Now $74">$74</div>
<div class="price value" aria-label="Now $75">$75</div>
<div class="discount-price" aria-label="Now $76">$76</div>
<div class="something_other">other</div>
'''

soup = BeautifulSoup(html_text, 'html.parser')

def is_price(tag):
    for k, v in tag.attrs.items():
        if 'price' in v:
            return True
        elif isinstance(v, list) and any('price' in i for i in v):
            return True


for tag in soup.find_all(is_price):
    print(tag)

Prints:

<div aria-label="Now $74" class="css-2vqe5n esdkp3p0" data-automation="buybox-price">$74</div>
<div aria-label="Now $75" class="price value">$75</div>
<div aria-label="Now $76" class="discount-price">$76</div>
user_100359850
Related