I use scrapy for a long time but now I need to use scrapy-splash for some reason.
I ran
docker run -it -p 8050:8050 --rm --name spider -v /etc/splash/proxy-profiles:/etc/splash/proxy-profiles scrapinghub/splash --max-timeout 3600
In /etc/splash/proxy-profiles/default.ini I write corporate proxy
Log:
[user@vir ~]$ sudo docker run -it -p 8050:8050 --rm --name spider -v /etc/splash/proxy-profiles:/etc/splash/proxy-profiles scrapinghub/splash --max-timeout 3600
2020-10-30 10:36:19+0000 [-] Log opened.
2020-10-30 10:36:19.324303 [-] Xvfb is started: ['Xvfb', ':1087332563', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-splash'
2020-10-30 10:36:19.496380 [-] Splash version: 3.5
2020-10-30 10:36:19.593826 [-] Qt 5.14.1, PyQt 5.14.2, WebKit 602.1, Chromium 77.0.3865.129, sip 4.19.22, Twisted 19.7.0, Lua 5.2
2020-10-30 10:36:19.594291 [-] Python 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]
2020-10-30 10:36:19.594536 [-] Open files limit: 1048576
2020-10-30 10:36:19.594911 [-] Can't bump open files limit
2020-10-30 10:36:19.627214 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2020-10-30 10:36:19.627645 [-] memory cache: enabled, private mode: enabled, js cross-domain access: disabled
2020-10-30 10:36:19.880659 [-] verbosity=1, slots=20, argument_cache_max_entries=500, max-timeout=3600.0
2020-10-30 10:36:19.881199 [-] Web UI: enabled, Lua: enabled (sandbox: enabled), Webkit: enabled, Chromium: enabled
2020-10-30 10:36:19.882106 [-] Site starting on 8050
2020-10-30 10:36:19.882406 [-] Starting factory <twisted.web.server.Site object at 0x7fda701815f8>
2020-10-30 10:36:19.883106 [-] Server listening on http://0.0.0.0:8050
To find out my docker ip: ip addr show docker0 | grep -Po 'inet \K[\d.]+'
It is 172.17.0.1
I try
curl 'http://172.17.0.1:8050/render.html?url=https://www.google.com/' --noproxy "*"
This --noproxy "*" is due to a company works behind a proxy.
And I get the 200 response in Docker Log. It says that proxy I set in /etc/splash/proxy-profiles/default.ini works correct.
2020-10-30 13:15:25.586864 [-] "172.17.0.1" - - [30/Oct/2020:13:15:25 +0000] "GET /render.html?url=https://www.google.ru HTTP/1.1" 200 199991 "-" "curl/7.29.0"
2020-10-30 13:18:22.103537 [events] {"path": "/render.html", "rendertime": 0.7088587284088135, "maxrss": 214808, "load": [0.02, 0.13, 0.14], "fds": 57, "active": 0, "qsize": 0, "_id": 140576160225768, "method": "GET", "timestamp": 1604063902, "user-agent": "curl/7.29.0", "args": {"url": "https://www.google.ru", "uid": 140576160225768}, "status_code": 200, "client_ip": "172.17.0.1"}
2020-10-30 13:18:22.103956 [-] "172.17.0.1" - - [30/Oct/2020:13:18:21 +0000] "GET /render.html?url=https://www.google.ru HTTP/1.1" 200 199128 "-" "curl/7.29.0"
Great! That's what I wanted
But when I try to implement ScrapyRequest and run Scrapy with code below
yield SplashRequest(
args={
'wait': 0.5,
'timeout': 10,
'proxy': '<hidden for security reasons>',
},
splash_url="http://172.17.0.1:8050",
url="http://www.google.ru/",
endpoint='render.html',
method="GET",
callback=self.parse_splash,
headers=headers,
dont_filter=True,
)
I got error
2020-10-30 16:45:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:45:37 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 1 times): 504 Gateway Time-out
2020-10-30 16:46:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:46:39 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 2 times): 504 Gateway Time-out
2020-10-30 16:47:27 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force
2020-10-30 16:47:27 [scrapy.core.engine] INFO: Closing spider (shutdown)
2020-10-30 16:47:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:47:40 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 3 times): 504 Gateway Time-out
2020-10-30 16:47:40 [scrapy.core.engine] DEBUG: Crawled (504) <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (referer: None)
Important note!
When I use just requests - it works!
When I use Scrapy requests - it works!
When I use ScrapySplash requests - it doesnt work!
Here is overriden settings of my Scrapy crawler:
custom_settings = {
'ITEM_PIPELINES': {
'spidermon.pipelines.DotcomPipeline': 400
},
'DUPEFILTER_DEBUG': True,
'HTTPERROR_ALLOWED_CODES': [503, 504],
'LOG_ENABLED': True,
'DOWNLOAD_DELAY': 0.5,
'DOWNLOADER_MIDDLEWARES': {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
},
'SPIDER_MIDDLEWARES': {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
},
'SPLASH_URL': "http://172.17.0.1:8050",
'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage',
}