When I run this script with Django shell it will run without error, but when I run with celery worker it get this error: OSError: [Errno 9] Bad file descriptor. If I remove channels from INSTALLED_APPS the problem will be solved and task runs with celery worker without error. Running the same script with python Process (due to ReactorNotRestartable exception) in django shell get the same error which celery worker had: OSError: [Errno 9] Bad file descriptor. I think there's a conflict between Django Channels & Scrapy when spider is executed in another process.
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
def run_spider(spider)
settings = get_project_settings()
crawler_process = CrawlerProcess(settings=settings)
crawler_process.crawl(spider)
crawler_process.start()
exception stack trace when run with celery worker:
Task run spider raised unexpected: OSError(9, 'Bad file descriptor')
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 256, in _add_reader
key = self._selector.get_key(fd)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py", line 192, in get_key
raise KeyError("{!r} is not registered".format(fileobj)) from None
KeyError: '9 is not registered'
exception stack trace when run with python Process in Django shell:
/usr/local/lib/python3.7/site-packages/billiard/pool.py in apply(self, func, args, kwds)
1387 '''
1388 if self._state == RUN:
-> 1389 return self.apply_async(func, args, kwds).get()
1390
1391 def starmap(self, func, iterable, chunksize=None):
/usr/local/lib/python3.7/site-packages/billiard/pool.py in get(self, timeout)
1787 return self._value
1788 else:
-> 1789 raise self._value.exception
1790
1791 def safe_apply_callback(self, fun, *args, **kwargs):
OSError: [Errno 9] Bad file descriptor