popups/modals/dialog window data scraping

user_14537553 0
Viewed 157

Is it possible to scrape data from popups, modals or dialog windows?

for example

https://tenders.procurement.gov.ge/public/?lang=en

I need email addresses from the users>suppliers and you can see users list, but I must open popup window in order to check any info about user.

So how could I scrape all emails from these popups ? is it possible?

first screen:
enter image description here

second screen:
enter image description here

third screen:
enter image description here

beautifulsoup python jquery javascript
1 Answers

You can use requests.session for the task:

import re
import requests
from bs4 import BeautifulSoup

base_url = 'https://tenders.procurement.gov.ge/public/?lang=en'
url = 'https://tenders.procurement.gov.ge/public/library/controller.php?action=org_list'
profile_url = 'https://tenders.procurement.gov.ge/public/library/controller.php?action=profile&org_id='
num = re.compile(r'(\d+)')

with requests.session() as s:
    
    # load cookies:
    s.get(base_url)
    
    soup = BeautifulSoup(s.get(url).content, 'html.parser')
    
    for tr in soup.select('tr[onclick]'):
        n = num.search(tr['onclick']).group(1)
        soup2 = BeautifulSoup(s.get(profile_url + n).content, 'html.parser')
        email = soup2.select_one('td:contains("E-Mail") + td')
        print(email.text)

Prints:

xxx@yandex.ru
xxx@gmail.com
xxx@gmail.com

...and so on.
user_100359850
Related