Python download a full web page (Including CSS)

Viewed 1396

I have a script running with Scrapy and Splash which pulls various data from a site, but am now trying to find a way of making an offline copy of each page too.

I have tried the following with Scrapy:

def parse(self, response):
    filename = response.url.split("/")[-1] + '.html'
    with open(filename, 'wb') as f:
        f.write(response.body)

And this with Urllib:

def parse(self, response):
    url = response.url
    file_name = 'aaa.html'
    with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
        shutil.copyfileobj(response, out_file)

Both of which work fine, but only download the HTML to file. I require the full page (with CSS included) - so is there any way of doing this, similar to replicating saving as 'Web Page, Complete' when manually saving sites using google chrome?

I suspect something like Selenium could, but guess it would be too slow for my application (which requires 100's, sometimes 1000's of URL's). Thanks.

1 Answers
Related