Scrape a website that loads table based on dynamically selected dropdown values

silentcobra 0
Viewed 18

I am trying to scrape table values from this website.. There are multiple dropdown which change based on selections in the previous dropdown. Upon inspecting the source, it seems to sending HTTP requests and render the results. However, I can't seem to write script to send those requests myself and scrape the data.

This is what I tried:

import requests

URL = 'https://voterlist.election.gov.np/bbvrs1/index_process_1.php' #API URL
payload = 'vdc=5298&ward=1&list_type=reg_centre' #Unique payload fetched from the network request
response = requests.post(URL,data=payload,verify=False) #POST request to get the data using URL and Payload information
print(response.text)

This is not giving me the expected response which should be a table containing the values. What could be the best approach to take in this case?

scrape python-requests python
1 Answers

So after a few hours of digging in I found an answer. All I had to do was send a cookie while making the request and also change the format the payload was sent in.

headers = {
     
    'num': number, # This the cookie that should be sent which happens to be a session id in my case. 
}

payload = {
    'state': "1",
    "district": "14",
    "vdc_mun": "5132",
    "ward": "3",
    "reg_centre" :"" 
    
}
#Then send a post request to the url
res = session.post(url, data=payload, headers=headers, verify=False)
silentcobra0
Related