Can't parse tabular content from some search results using post requests

Question

I'm trying to fetch some tabular content from a webpage using the script below. To populate the content manually, it is necessary to choose the options from the dropdown shown in this image before hitting the Submit button. I've tried to mimic the post http requests accordingly. However, I might have gone somewhere wrong and which is why the script is not working. To be specific, this is what I'm trying to fetch.

This is how I've tried:

import requests
from bs4 import BeautifulSoup

URL = 'https://www.lgindiasocial.com/microsites/brand-store-web-five/locate.aspx'

headers = {
    'x-microsoftajax': 'Delta=true',
    'origin': 'https://www.lgindiasocial.com',
    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'referer': 'https://www.lgindiasocial.com/microsites/brand-store-web-five/locate.aspx',
}

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'
    r = s.get(URL)
    soup = BeautifulSoup(r.text,"lxml")
    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['ScriptManager1'] = 'UpdatePanel1|btnsubmit'
    payload['ddlState:'] = 'Assam'
    payload['ddlCity'] = 'Golaghat'
    payload['ddllocation'] = 'Golaghat'
    s.headers.update(headers)
    r = s.post(URL,data=payload)
    soup = BeautifulSoup(r.text,"lxml")
    item = soup.select_one("table")
    print(item)

When I run the script, I get None as output.

How can I fetch the tabular content from the search results using post requests?

EDIT: If I copy the content of payload directly from dev tools and use the same within payload I get desired results.

import requests
from bs4 import BeautifulSoup

URL = 'https://www.lgindiasocial.com/microsites/brand-store-web-five/locate.aspx'

payload = "ScriptManager1=UpdatePanel1%7Cbtnsubmit&hidcity=&ddlState=Assam&ddlCity=Golaghat&ddllocation=Golaghat&__EVENTTARGET=&__EVENTARGUMENT=&__LASTFOCUS=&__VIEWSTATE=M%2BqldpZhV90EX2sawXMrHD7jYtOMXnrPuP8XfVtS21GKmxK0YYuBnqm3I7tU%2BKMtFGZgzWpsYK%2FYJtfTBUK%2F0WobR21tjbWjdrZiXS5FlLcS6qgYMNKfqyZRcK13dbz667H7T6QZqpITTRSqsM%2BrM91VW989KXoknFdx0H6EkRFCJRu4WsBsUxeJnd5Lf5IAUN%2BTNKDYE5GuclDNKnmU1pMmHhrjKQysvYtw8cjD5DdDkNb7NDkLiVxm7DISyXZtVJyOBV6dFa%2Blm1%2FR9M7F2nyepARAl0XIiNP9dhFvomLNdlP%2BU%2FNyllJ5IXW4D%2Fl5Kfx5yaRP8XSKURtAc915i%2F2T48a0dyAR42tJ40eit1IWs7MCwgesNtF35zkuKN1SRhyhHqcnKjcMYW%2BkLqKsLvKpLQcDuXrIAzYyqlgJZ%2FlBQJo%2BiM4tTOH4mEqDkSZW%2Fk94KX1OM70s9%2FS%2Fd5trrHIgNoKw1bCRI8IQ41ZEicMsJPTp67KnqoMZz0F0cCmo%2F49zYkuHw0kqaZmKCrRUNW8Xcr%2F5A3AfNg%2FB8WURD0g2x%2BwzcLXDcVCJ6ngf0LdOc%2BTppM6EOZpTGJGjjDqK116tzWAOPfiJHgBuIPkiZJTaEHnwwjcYXuuLN%2FTgPFUJkXVjBSyRdCnPXsebInNd4Wsu2lnNdwZUO3rnNuu5eY%2FHf7YemcmCEzji%2FxLG%2FynnG0sG61TC1bJCyFw2E3V6ZGshbuqDfh7QQyxqPDEt2uaCN7s%2FOZ%2FwiXeVY2henUVBZSVrxUvF6QT0eO4SIY0OlNYBLK7cO4YG4zC0tURSBr7lZwR%2B%2FowLieNGSO7sOeLQVwL71GKnzBAOZVQH1hw%2B8FIRPoc0pn3v7RjK5CMgTtrZlar67Cv1lTi2nUyAIpX%2BhGkaQeOsg%2ByaIqDIo%2FWwcrg9VV9QP%2FdmwP8hTtq3KTVs0Ncja4Yvizm12BkEwWtMJ9fqzLBXt%2F2J2EjsG7GudgXypwSU7U8oY%2Fq%2BCk93y%2FeTr1ftEFbpGRTRm4hNVXeoCYRyuJceU%2BvO4U5E29ZPqBIolidYtKKH7lnRxKNk2BHtY93VNHPZEjTEDnHcGbgtHmxlBjHRQZlzJKWTjY5ccdFABihGx%2FzY0VCwaehpx2BWxy5qXqW1fX7e5uxxxHteYVt7YyrzYPsX%2B%2FlKiYwt23fsJzmmVkHwmu5%2FTSk1Ms9yJmBE%2B8pEF%2Bum01L8jRH4zxyTaD4s779uLZwLAUUzpi5cfseKTrjGv7uNjCpNci9BXbSdCdqrKa8aPiJX0lWUH9zid%2B8Jc7Jhx%2Bb6nzJpbZ8E9sPpUlcHVGUSzqixsiK91W%2FDDk2LCOvTqJJ9JXmy5cwRhL9r95okWq%2BDImTetFhdYk9%2F9VH3JsACpv4dqqdviEjjFpvmEp7SBMLSWw7toPUIRortPtriz3u9velTqNpHgmbmig8Znb%2F4Q8JrYfjPZzfRxN%2FuQXQyxUNUY2IsYbC5Bm7JWTMZe869muBdE%2FlMLujUkOFCXaOwZXuZHbr7neq0nro3RvYUggBLqxGFlG1Bp52iDNklcx8nfjVMOhOybfCMcxz6mq4Ew2hdLv4IslLRawI5u%2FPQe0vu0TG9LeBeR6Ok1sf72rWpvhD6yl4GTy8oJC1UglabWo8i5aMprxxAWuz%2BzLzizI3aRTQsl1MFKsD9gIGZsaFNAIb7gEgFgw%2B%2BSjTGR51mGES3sOUYXscIJVBciBs3F9vnr8u5gfKD3hLwqvc4djKMBxVQfjLEs%2FQwb7mlOx8XodaV6uOrkiZpw2WZNja5RPBIp4VXeXKXIxqBNsNA4eGT%2Bx2b2JadVB8%3D&__VIEWSTATEGENERATOR=06ED1D24&__VIEWSTATEENCRYPTED=&__ASYNCPOST=true&btnsubmit=Submit"

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'
    s.headers.update({'content-type':'application/x-www-form-urlencoded; charset=UTF-8'})
    r = s.post(URL,data=payload)
    soup = BeautifulSoup(r.text,"lxml")
    item = soup.select_one("table")
    print(item)

user_8032644 · Answer

First a small typo in your code, (extra colon in there)

payload['ddlState:'] = 'Assam'

The larger problem is to do with the way the page is constructed. The page has three dropdowns, and those dropdowns send a POST request. Each of the POST requests returns a modified __VIEWSTATE that needs to be included in the header of the subsequent request.

In your code, you are taking the __VIEWSTATE from the input[form] on the original GET request only, you need to get the __VIEWSTATE from the last POST request. So the following should work:

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'
    r = s.get(URL)
    soup = BeautifulSoup(r.text, "lxml")

    # first POST = Select State
    payload = {i['name']: i.get('value', '') for i in soup.select('input[name]')}
    payload['ScriptManager1'] = 'UpdatePanel1|btnsubmit'
    payload['ddlState'] = 'Assam'
    payload['ddlCity'] = 'Select City'
    payload['ddllocation'] = 'Select Location'
    payload['__EVENTTARGET'] = 'ddlState'
    r = s.post(URL, data=payload)
    soup = BeautifulSoup(r.text, "lxml")

    # second POST = Select City
    payload = {i['name']: i.get('value', '') for i in soup.select('input[name]')}
    payload['ScriptManager1'] = 'UpdatePanel1|btnsubmit'
    payload['ddlCity'] = 'Golaghat'
    payload['ddllocation'] = 'Select Location'
    payload['__EVENTTARGET'] = 'ddlCity'
    r = s.post(URL, data=payload)
    soup = BeautifulSoup(r.text, "lxml")

    # third POST = Select Location
    payload = {i['name']: i.get('value', '') for i in soup.select('input[name]')}
    payload['ScriptManager1'] = 'UpdatePanel1|btnsubmit'
    payload['ddlCity'] = 'Golaghat'
    payload['ddllocation'] = 'Golaghat'
    payload['__EVENTTARGET'] = ''

    s.headers.update(headers)
    r = s.post(URL, data=payload)
    soup = BeautifulSoup(r.text, "lxml")
    item = soup.select_one("table")
    print(item)

There's room for some optimisation of this code. I tried to make the problem transparent.

user_2823719 · Answer

The result coming back from your final post is shown below. It does not contain your expected table not is it a complete HTML document. I suspect this is normally loaded using AJAX so that only a portion of the DOM is updated and then subsequent to this being processed more JavaScript is executed to load the table you seek. My suggestion to you is to use a tool such as selenium to drive your browser. For example, using Chrome:

from selenium import webdriver
from selenium.webdriver.support.select import Select
from bs4 import BeautifulSoup

from selenium.common.exceptions import StaleElementReferenceException
import time

class WaitUntilElementIsStale:
    def __init__(self, *, driver=None, element=None, timeout=10):
        assert driver or element
        self.element = driver.find_element_by_tag_name('html') if element is None else element
        self.timeout = timeout

    def __enter__(self):
        return None

    def __exit__(self, exc_type, exc_value, exc_traceback):
        if exc_type is not None:
            return
        start_time = time.time()
        while time.time() < start_time + self.timeout:
            try:
                # poll the link with an arbitrary call
                self.element.find_elements_by_id("doesn't-matter")
            except StaleElementReferenceException:
                return
            time.sleep(0.1)


options = webdriver.ChromeOptions()
options.add_argument("headless") # comment out to see the browser
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
try:
    driver.implicitly_wait(10) # 10 seconds is a bit of "overkill" but it doesn't hurt

    driver.get('https://www.lgindiasocial.com/microsites/brand-store-web-five/locate.aspx')
    with WaitUntilElementIsStale(element=driver.find_element_by_name("ddlCity")):
        Select(driver.find_element_by_name("ddlState")).select_by_value('Assam')
    with WaitUntilElementIsStale(element=driver.find_element_by_name("ddllocation")):
        Select(driver.find_element_by_name("ddlCity")).select_by_value('Golaghat')
    Select(driver.find_element_by_name("ddllocation")).select_by_value('Golaghat')
    driver.find_element_by_name('btnsubmit').click()
    driver.find_element_by_tag_name('table') # implicitly wait for up to 10 seconds for this element to appear
    soup = BeautifulSoup(driver.page_source, "lxml")
    item = soup.select_one("table")
    print(item)
finally:
    driver.quit()

Prints:





LG BRAND STORE - BRAHMAPUTRA ELECTRONICS
Complete Address : 
g.f. road, near das & co., golaghat, assam

Contact Person :
pravin kankani

Contact No. : 9954305770 , 

 Email ID: brahmaputralgshoppe@yahoo.com

UPDATE:

I see that you have updated your question. I assume that you have done a network trace to see what the subsequent AJAX POST request is being used to get the final results. You are very dependent on the internals of the website's application not changing, but I suppose to the extent that they remain constant your code will work.

I have updated my solution to not have to arbitrarily wait 1 second each time a selection is made in a dropdown before attempting to make a selection in the next dropdown. After a selection is made, for example, in the state dropdown, the current city dropdown is recreated (and the original one becomes "stale"). The new code just repeatedly waits for the current city dropdown to become stale before looking for the new one. Thus, there should be little waste of time.

The second change is simply to load the complete HTML into Beautiful Soup as you do.

Of course, this code is dependent on the user interface not changing. But it is rather apparent when the user interface does change and this code is easily adapted to such changes.

Results of Your Original Post Request:

3256|updatePanel|UpdatePanel1|
      
                
                        Locate nearest LG Brand Store
                        
                                
                                        
                                        
                                        
                                        
                                
                        
                        
                                
                                        
                                        
                                                
                                        
                                        
                                                
                                                                


                                                
                                        
                                        
                                        
                                
                        
                
          
      |0|hiddenField|__EVENTTARGET||0|hiddenField|__EVENTARGUMENT||0|hiddenField|__LASTFOCUS||1648|hiddenField|__VIEWSTATE|9aji4H+sHZgnrG7tl/cn055Jcu74N6ZhHGoSzOvQqlBrKXUD9/cBQ156dQZjy+TjfXw0Fkg3939tdE4izxjfd/hvGfV4nC0wmZIv3wKCnQdxKzH0RFeDbEqZ6m1kn1GeiQCcDTOQ0UySTp+gTFcA7Jih/UbFnTjOsDJ7tk1JEMlsw7O8Ss/g1Fvc/Vx1Oh4xVEvtS0hlmp6WbU2dpm8dMvVdcy1Or8IgSDoVoNQbqVDXcRkZEPHk9uV4p/VuoMU9LgSDVlrd9FCxV5rP4xkW1+THBX1FqCUYQfk8VfayTQTg1FyQfs0lWHMpXmBNpUhSGxXkLL6GQ9s4dNzKcJa8NWcwjTu78b168nL3TB2aw0iVARxkIUXOn+E51MFFfzdDJYgBhar7XJiF9AUQh2k4wwgTB4FmNGxlO9bGtGZMdB7NR19OmLoP3Tz1xZ9wJaJI2rBdbGGvpjUJHlVnoAx9Pn8jmzVtuqA/KFmgJZd2XfQAlez8qXZz/Nakl22uhzgCZDdl77H25PU4aZhHemR8Ut4sYNposzdkUuD1FxVlTfkqRcFMVpSdLWpQZeowAJx12wbv/ny8ohsEMuTs7dfhAqXw01HlEh22SXoWDzrTeeFPNn2ryqyloczBleGoXDHqeAAygeIHmnjM1eHujrcIER5dijk1cU7OH7FpRyBp3ACvvZSprRAQBy4ORiu5F8LBbPiTPVxWwFjNijL2hmW9yCia54tLdaSmx3zNlQXjel9Mo2D9QhrMs45p4srk2iFuItuUCdiA+oCsb+BHGP5WhypWYR/z6WDnY8fjBnRkIxrLNehve9qN2DLtAZGoO+UuVmqPooMBTv0RPw9cHdF52Yu52ZtHtFat7BTH2vrneUsIsEBzMc19/NBswNuugvJQDJ/OHUEx1f7uB6cdqFTUyTc+LOK8kW5lZMKyUzS4Bzc5g5XIi4e9c9/B1Bgvj16aT8oKyryK92uhjOrMNP5uSh/izGBB3WH1semoIih72Jy8ydLjOsk3qSylCqoYzQSSCTXRXeCZcengG4dgFTVwW32wA7WjbpUGZw1/YKENhiesda8ZP/nrFNmXDFqMHNm5Oe059jCxk+GpsKRRK3eN5nnXTez3uXx+XdJNrgVBR+gufhsAW/Z7N4b1SKpBnKa3t7O7KBUhrn60PnluMlmrAq6yppcJ6SSyBLVbJPTZLORUKmp2ha3FUSRBl/Ei8bx1FdJXFFcWP0k/c+vWmsBpiyvkD6azYWadVRcL6OdTb9iBl2BP4qktMGxQj7xtFbP8Jk3iAbf/CULABw6wUrl+JdCITSNPfypfsM+MWFKEb1M4a+PGtRcYBIcWe7hmzyjau9j1XCxZJuUNMgiwQYv/Zdl8pMzKEFOM9Rt9oUQvkOnOVLggqqK+rLhiSbwZfI7HRvoC3eLdta9NY7D+6PNrwGkfOnnjCeyWTRMHD61rA2tt/bHOJNr+HEgIMimYgsfMW6q244E/OLDDLPCYcBvHpDsJilHJCY9olAm6+j/8jAyYzHvDzUJSDpf6PImydtv7er2gvryRRBAeImlSLasgP+3ujnrfgG3DA9L6PoV2P9Ft6INQGhrHIwmxtCUnl7THLHCLcqFGW6mG4Sn5CJTp2BnqtM0xFecL/ytDHA+xYHnEs4ky|8|hiddenField|__VIEWSTATEGENERATOR|06ED1D24|0|hiddenField|__VIEWSTATEENCRYPTED||0|asyncPostBackControlIDs|||0|postBackControlIDs|||13|updatePanelIDs||tUpdatePanel1|0|childUpdatePanelIDs|||12|panelsToRefreshIDs||UpdatePanel1|2|asyncPostBackTimeout||90|11|formAction||locate.aspx|14|scriptBlock|ScriptContentNoTags|var player=[];|13|scriptStartupBlock|ScriptContentNoTags|initialize();|

Can't parse tabular content from some search results using post requests

2 Answers