Python (bs4 + selenium) - Search html data after simulated some actions with selenium

user_13924483 0
Viewed 74

I wrote the following code:

driver = webdriver.Firefox()
driver.get('https://www.ybpn.de/ihre-parfuemerien/')

search_elem = driver.find_element_by_id("search_input") # Find search form
search_elem.send_keys("80331") # Enter zip code - later based on a .txt of zip codes
submit_button = driver.find_element_by_name("tx_ybpn_storefinder[submit]").submit() # press "submit" button, search

Selenium opens the url, search for the "input field", enter a zip code and press submit.

Now I want to search the "new page" (after the action from the code above) for specific data with bs4.

Normally I would use the following code:

url = ("https://www.ybpn.de/ihre-parfuemerien/")
page = requests.get(url)
soup = BeautifulSoup(page.text, features="html.parser")

Problem: When I use the "normal" url it loads the dynamic map without the data I need. The page doesnt create a unique url after I performed the search with selenium. It only adds a specific cHash-value at the end of the url, but when I connect with this url, Im redirected to the "normal page" - there is no url that loads the page with my search results.

Question: How can I get a specific url or create a variable after the "search for specific zip code"-action to search with bs4 for the data I need?

I really hope its understandable what I mean! Thank you!

selenium request beautifulsoup python
2 Answers

To get contacts for specific ZIP code, you can use this example:

import requests
from bs4 import BeautifulSoup


url = 'https://www.ybpn.de/ihre-parfuemerien/'
data = {'tx_ybpn_storefinder[searchReq][term]': '80331'}

soup = BeautifulSoup(requests.get(url).content, 'html.parser')
data['tx_ybpn_storefinder[__trustedProperties]'] = soup.select_one('#storefinder [name*=__trustedProperties]')['value']
soup = BeautifulSoup(requests.post(url, data=data).content, 'html.parser')

for item in soup.select('.storefinder__list-item'):
    print(item.select_one('.storefinder-item__title').get_text(strip=True))
    print(item.select_one('.storefinder-item__adress').get_text(strip=True, separator='\n'))
    print(item.select_one('.storefinder-item__contact').get_text(strip=True))
    print('-' * 80)

Prints:

Parfümerie Brückner
Rindermarkt 1
80331
München
Tel.:+49 89 263181
--------------------------------------------------------------------------------
Parfümerie Brückner
Marienplatz 8
80331
München
Tel.:+49 89 223874
--------------------------------------------------------------------------------
City Parfümerie Rathjen
Tal 14
80331
München
Tel.:+49 89 2285222
--------------------------------------------------------------------------------
Die kleine Theatiner Parfümerie
Theatiner Straße 35
80333
München
Tel.:+49 89 267919
--------------------------------------------------------------------------------
City Parfümerie Rathjen
Stachus Passagen Karlsplatz 1. UG
80335
München
Tel.:+49 89 55 4009
--------------------------------------------------------------------------------

...and so on.
user_100359850

I have an idea based on the accepted answer, which is to get the inner HTML of the whole body.

    driver = webdriver.Firefox()
    driver.get('https://www.ybpn.de/ihre-parfuemerien/')
    
    search_elem = driver.find_element_by_id("search_input") # Find search form
    search_elem.send_keys("80331") # Enter zip code - later based on a .txt of zip codes
    submit_button = driver.find_element_by_name("tx_ybpn_storefinder[submit]").submit() # press "submit" button, search

soup = BeautifulSoup(page.find_element_by_tag_name("body").get_attribute('innerHTML'), features="html.parser")
for item in soup.select('.storefinder__list-item'):
    print(item.select_one('.storefinder-item__title').get_text(strip=True))
    print(item.select_one('.storefinder-item__adress').get_text(strip=True, separator='\n'))
    print(item.select_one('.storefinder-item__contact').get_text(strip=True))
    print('-' * 80)
user_107311200
Related