scrapy how to load urls from file at scrapinghub

Viewed 1004

I know how to load data into Scrapy spider from external source when working localy. But I strugle to find any info on how to deploy this file to scrapinghub and what path to use there. Now i use this approach from SH documentation - enter link description here but recieve NONE object.

import pkgutil

class CodeSpider(scrapy.Spider):
name = "code"
allowed_domains = ["google.com.au"]

def start_requests(self, ):

    f = pkgutil.get_data("project", "res/final.json")
    a = json.loads(f.read())

Thanks. My setup file

from setuptools import setup, find_packages

setup(
    name         = 'project',
    version      = '1.0',
    packages     = find_packages(),
    package_data = {'project': ['res/*.json']
    },
    entry_points = {'scrapy': ['settings = au_go.settings']},

    zip_safe=False,

)

The error i got.

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "/tmp/unpacked-eggs/__main__.egg/au_go/spiders/code.py", line 16, in start_requests
    a = json.loads(f.read())
AttributeError: 'NoneType' object has no attribute 'read'
1 Answers
Related