Is it possible to trigger Google Maps api's JavaScript click listener then scrape the data via using Python?

user_8696220 0
Viewed 113

http://ihe.istanbul/satis-noktalari

I would like to scrape the points(latLng) data of the targeted company's dealerships on maps which uses Google Maps api.

I tried to scrape data by using requests_html to render JavaScript on the page of website, then I used to reach the element by using BeautifulSoup.

from bs4 import BeautifulSoup
from requests_html import HTMLSession

# create an HTML Session object
session = HTMLSession()

# Use the object above to connect to needed webpage
resp = session.get("http://ihe.istanbul/satis-noktalari")

# Run JavaScript code on webpage
resp.html.render()

soup = BeautifulSoup(resp.html.html, "lxml")

html_content = soup.contents[1]

_script = html_content.find_all("script")[23]
print(_script)

Therefore, the print lead me to a way that I can see the desired area where I can find the latLng point values if the click event is triggered.

However, the website's url cannot renew itself and put a tag for selected zone of the city.

To explain myself in a clear way, I created two pictures that show exactly what I want to do:

This output shows the result which there is no selected city:

This output shows the result which there is no selected city

This is the triggered click event which shows the desired result:

This is the triggered click event which shows the desired result

If url could be updated after trigger JavaScript event via Google Maps api, I could use the url. How can I trigger it via using Python, or how can I scrape the triggered data by using Python? The Python code I provide shows non triggered event.

web-scraping beautifulsoup google-maps python javascript
1 Answers

Given that the page dynamically issue POST XHR requests using the options value attribute value, within the select dropdown. You can extract those values, mimic the POST requests page does, then use regex to extract the lat, lon from the responses. The following shows the logic for grabbing the centre specified co-ordinates.

import requests, re

headers = {
    'Content-Type': 'application/x-www-form-urlencoded',
    'User-Agent': 'Mozilla/5.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Referer': 'http://ihe.istanbul/satis-noktalari',
    'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
}

results = {}

with requests.Session() as s:
    
    s.headers = headers
    r = s.get('http://ihe.istanbul/satis-noktalari')
    soup = bs(r.content, 'lxml')
    options = {i.text:i['value'] for i in soup.select('[name=ilceID] option:nth-child(n+2)')} 
    
    for k, v in options.items():

        data = {'ilceID': v, 'SatisBufe': '1'}

        r = s.post('http://ihe.istanbul/satis-noktalari', data=data)
        
        lat, lon = re.search(r'google.maps.LatLng\(([\d.]+),\s?([\d.]+)\)', r.text).groups()
        
        print(k, f'lat = {lat}', f'lon = {lon}')
        
        results[k] = [lat, lon]
user_62412350
Related