Start web scraper from specific aspx page

Viewed 377

I am trying to scrape data from this webpage. It is a big job -- there are ~600 next page links, and the scraper breaks down after 300 pages (8 hours) because of a "connection reset by peer" error.

Rather than starting fresh after each crash, I want the scraper to start at page 300 and hopefully it will reach the last page without error and I can append the two outputted data files. My code (below) works when I go through the pagination consecutively (starting at page 1), but it doesnt work if open page 1 and try to post to page 300. I get an error about the page_no variable "AttributeError: 'NoneType' object has no attribute 'find' " which implies it never reached page 300. Any ideas what the issue is and how to solve it?

#Open Search Page
url = 'http://forestsclearance.nic.in/'            
r = requests.get(url + 'Online_Status.aspx')
VIEWSTATE, GENERATOR, VALIDATION = getFormData(r.content)
cookies = {
    'ASP.NET_SessionId': 'kaqs1jzegnfn4zxpwio4jthl',
    'countrytabs': '0',
    'countrytabs1': '0',
    'acopendivids': 'Omfc,Email,Campa,support,livestat,commitee,Links',
    'acgroupswithpersist': 'nada',
}
headers = {
    'Connection': 'keep-alive',
    'Cache-Control': 'max-age=0',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
}
#Click Search box
r = requests.post(
    url + 'Online_Status.aspx',
    headers=headers,
    cookies=cookies,
    data = {
        'ctl00$ScriptManager1': 'ctl00$ContentPlaceHolder1$UpdatePanel1|ctl00$ContentPlaceHolder1$Button1',
        '__EVENTARGUMENT': '',
        '__EVENTTARGET': '',
        '__VIEWSTATE': VIEWSTATE,
        '__VIEWSTATEGENERATOR': GENERATOR,
        '__VIEWSTATEENCRYPTED': '',
        '__EVENTVALIDATION': VALIDATION,
        'ctl00$ContentPlaceHolder1$ddlyear': '-All Years-',
        'ctl00$ContentPlaceHolder1$ddl1': 'Select',
        'ctl00$ContentPlaceHolder1$ddl3': 'Select',
        'ctl00$ContentPlaceHolder1$ddlcategory': '-Select All-',
        'ctl00$ContentPlaceHolder1$DropDownList1': '-Select All-',
        'ctl00$ContentPlaceHolder1$txtsearch': '',
        'ctl00$ContentPlaceHolder1$HiddenField1': '',
        'ctl00$ContentPlaceHolder1$HiddenField2': '',
        '__ASYNCPOST': 'false',
        'ctl00$ContentPlaceHolder1$Button1': 'SEARCH',
    }
)
VIEWSTATE, GENERATOR, VALIDATION = getFormData(r.content)

#Post to Page 300
lastPage = 563
for page in range(300, lastPage + 1):
    r = requests.post(
    url + 'Online_Status.aspx',
    cookies=cookies,
    data = {
    'ctl00$ScriptManager1': 'ctl00$ContentPlaceHolder1$UpdatePanel1|ctl00$ContentPlaceHolder1$Button1',
    'ctl00$ContentPlaceHolder1$RadioButtonList1': 'New',
    '__EVENTARGUMENT': 'Page${}'.format(page),
    '__EVENTTARGET': 'ctl00$ContentPlaceHolder1$grdevents',
    '__VIEWSTATE': VIEWSTATE,
    '__VIEWSTATEGENERATOR': GENERATOR,
    '__VIEWSTATEENCRYPTED': '',
    '__EVENTVALIDATION': VALIDATION,
    'ctl00$ContentPlaceHolder1$ddlyear': '-All Years-',
    'ctl00$ContentPlaceHolder1$ddl1': 'Select',
    'ctl00$ContentPlaceHolder1$ddl3': 'Select',
    'ctl00$ContentPlaceHolder1$ddlcategory': '-Select All-',
    'ctl00$ContentPlaceHolder1$DropDownList1': '-Select All-',
    'ctl00$ContentPlaceHolder1$txtsearch': '',
    'ctl00$ContentPlaceHolder1$HiddenField1': '',
    'ctl00$ContentPlaceHolder1$HiddenField2': '',
    '__ASYNCPOST': 'false',
        }
    )

    #scrape data
    soup = BeautifulSoup(r.content, 'lxml')
    table = soup.find('table', {'id' : 'ctl00_ContentPlaceHolder1_grdevents'})
    page_no = int(table.find('tr', {'class': 'pagi'}).span.text)
    rows = table.findAll('tr')
    for row in rows[1:len(rows)-2]:
        #My scraping code goes here...

    #Get form data for next page post request
    VIEWSTATE, GENERATOR, VALIDATION = getFormData(r.content)
1 Answers

it sounds like table (and probably soup, and maybe r.content) isn't the object you're expecting it to be (null?). Have you tried dumping the response headers on that request? If you can show us the results of print(r.headers) before you load your soup variable, we may be able to help you dig further.

Related