How to scrape web pages by using a search term without knowing the tags/classes?

user_7644562 0
Viewed 317

I'm working on a project using Python(3.7) and BeautifulSoup(4) to implement a scraping solution.

Note: I have searched to find a solution to my problem, but I couldn't find any solution because it's different from what usually we need for scraping. So, that's why, don't mark this as duplicate, please!

This project has two parts:

  1. We have grabbed the Google search results URLs (first 5 for example) on the base of a search term.
  2. Then, we have to scrape those search results' URLs to grab the search term's related information from those pages, so we don't know the actual classes/tags of those results' pages.

So, how we can grab the search term's relevant information from web pages without knowing the actual tags/classes?

Here's what I have done so far:

soup = BeautifulSoup(driver.page_source, 'lxml')
result_div = soup.find_all('div', attrs={'class': 'g'})

links = []
titles = []
descriptions = []
for r in result_div:
    # Checks if each element is present, else, raise exception
    try:
       link = r.find('a', href=True)
       title = None
       title = r.find('h3')

       if isinstance(title, Tag):
          title = title.get_text()

          description = None
          description = r.find('span', attrs={'class': 'st'})

       if isinstance(description, Tag):
           description = description.get_text()

       # Check to make sure everything is present before appending
       if link != '' and title != '' and description != '':
           links.append(link['href'])
           titles.append(title)
           descriptions.append(description)
    # Next loop if one element is not present
    except Exception as e:
        print(e)
        continue
scrapy web-scraping beautifulsoup python-3.x python
1 Answers

It is easy to find the elements that contain a keyword or a regex easily in an HTML string, this is how you can do it.

This will return every element in your HTML page that contains the keyword you wish to look for.

from bs4 import BeautifulSoup
import re

html_text = """
<h2>some other text</h2>
<p>text you want to find with keyword</p>
<h1>foo bar foo bar</h1>
<h2>text you want to find with keyword</h2>
<a href="someurl">No idea what is going on</a>
<div> text you want to find with keyword</div>
"""

soup = BeautifulSoup(html_text)


for elem in soup(text=re.compile(r'\bkeyword\b | \bkey_word\b | \something else\b | \bone_more_maybe\b')):
    print(elem.parent)
user_86613270
Related