ImportError: No module named bs4 scraping hub

Admin 1
Viewed 461
Packing version 1.0
Deploying to Scrapy Cloud project "316316"
Deploy log last 30 lines:
    _get_apisettings, commands_module='sh_scrapy.commands')
  File "/usr/local/lib/python2.7/site-packages/sh_scrapy/crawl.py", line 148, in _run_usercode
    _run(args, settings)
  File "/usr/local/lib/python2.7/site-packages/sh_scrapy/crawl.py", line 103, in _run
    _run_scrapy(args, settings)
  File "/usr/local/lib/python2.7/site-packages/sh_scrapy/crawl.py", line 111, in _run_scrapy
    execute(settings=settings)
  File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 149, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 251, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 137, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 338, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 61, in from_settings
    return cls(settings)
  File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 25, in __init__
    self._load_all_spiders()
  File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
    for module in walk_modules(name):
  File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/app/__main__.egg/niceprice/spiders/forever.py", line 11, in <module>
ImportError: No module named bs4
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

Hello, I wrote a scrapy spider, and I am trying to deploy onto scraping hub, I needed to use BeautifulSoup to handle broken HTML parts, I am getting the following error when deploying.

python
1 Answers

From support page on scrapinghub.com:

Here's what you have to do in order to deploy python dependencies for your projects:

  1. create a requirements.txt file containing your extra dependencies in your project's root folder
  2. set that file as the requirements value in your project's scrapinghub.yml configuration file
  3. deploy your project with the dependencies

1. Creating the requirements.txt file

This is a regular text file where you list the Python packages that your project depends on, one package per line. For example:

js2xml==0.2.1 
extruct==0.1.0 
requests==2.6.0

You should always pin down the specific versions for each of your dependencies, as we did in the example above. By doing this, you avoid the trouble of getting your spiders broken due to unexpected upgrades.

2. Configuring your dependencies on scrapinghub.yml

After creating the requirements file, add the requirements setting to scrapinghub.yml and point it to your project's requirements.txt path:

projects:
   default: 12345 
requirements:
   file: requirements.txt

Note: if there's no scrapinghub.yml file in your project folder, you should run shub deploy once to generate it.

3. Deploying your project

Now that you've set your project dependencies, it's time to deploy your project. Just run a regular shub deploy and then you should be able to use your spiders with extra dependencies in Scrapy Cloud.

$ shub deploy

user_40470830
Related