Downloading a file through a JS function with requests-html

Viewed 30

I am scraping a website where I need to download a file. The file is usually downloaded by clicking a button Generate, however looking into the source code it calls a JavaScript function that submits a form, it looks something like this:

function formFunc{
    // Declare a bunch of form values here
    document.frmCriterio.action = "main_xml.asp";
    document.frmCriterio.target = "_self";
    document.frmCriterio.submit();
    return true;
}

I tried to replicate the behavior by navigating to that page with python requests-html and using the render method to run that same function with my given parameters:

response.html.render(script=create_js(prov_id), cookies=cookies)

However when I try to do this I get an error:

Traceback (most recent call last):
  File "D:\PycharmProjects\sears_mvp\sears_session.py", line 280, in <module>
    my_func(param)
  File "D:\PycharmProjects\sears_mvp\sears_session.py", line 235, in my_func
    response.html.render(script=create_js(prov_id), cookies=cookies)
  File "D:\PycharmProjects\sears_mvp\venv\Scripts\lib\site-packages\requests_html.py", line 670, in render
    raise MaxRetries("Unable to render the page. Try increasing timeout")
requests_html.MaxRetries: Unable to render the page. Try increasing timeout

Even after changing the parameters for timeout, and retries as well as some others I still got the same error. I even tried changing the target for document.frmCriterio.target (see above), but to no avail.

I know the JS function itself is working because I have navigated to the site on Firefox, inserted the function via the console, called it from there and successfully downloaded the file. So the issue is with accessing it via requests-html.

So my question is what should I do to be able to download this file?

Whether it be to modify the JS function or to implement a solution in python in this module or another module or something completely different.

(PS: I want to avoid using Selenium at the moment, but I can migrate the project over the Selenium if I must.)

0 Answers
Related