Select css tags with randomized letters at the end

user_6943486 0
Viewed 67

I am currently learning web scraping with python. I'm reading Web scraping with Python by Ryan Mitchell.

I am stuck at Crawling Sites Through Search. For example, reuters search given in the book works perfectly but when I try to find it by myself, as I will do in the future, I get this link.

Whilst in the second link it is working for a human, I cannot figure out how to scrape it due to weird class names like this class="media-story-card__body__3tRWy"

The first link gives me simple names, like this class="search-result-content" that I can scrape.

I've encountered the same problem on other sites too. How would I go about scraping it or finding a link with normal names in the future?

Here's my code example:

from bs4 import BeautifulSoup
import requests
from rich.pretty import pprint

text = "hello"

url = f"https://www.reuters.com/site-search/?query={text}"

response = requests.get(url)
soup = BeautifulSoup(response.text, "lxml")

results = soup.select("div.media-story-card__body__3tRWy")

for result in results:
    pprint(result)
    pprint("###############")
web-scraping beautifulsoup css html python
1 Answers

You might resort to a prefix attribute value selector, like

div[class^="media-story-card__body__"]

This assumes that the class is the only one ( or at least notationally the first ). However, the idea can be extended to checking for a substring.

user_5889160
Related