I have a script running with Scrapy and Splash which pulls various data from a site, but am now trying to find a way of making an offline copy of each page too.
I have tried the following with Scrapy:
def parse(self, response):
filename = response.url.split("/")[-1] + '.html'
with open(filename, 'wb') as f:
f.write(response.body)
And this with Urllib:
def parse(self, response):
url = response.url
file_name = 'aaa.html'
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
Both of which work fine, but only download the HTML to file. I require the full page (with CSS included) - so is there any way of doing this, similar to replicating saving as 'Web Page, Complete' when manually saving sites using google chrome?
I suspect something like Selenium could, but guess it would be too slow for my application (which requires 100's, sometimes 1000's of URL's). Thanks.