can't run scrapy spider from a .bat file

Viewed 19

I've been trying to run a scrapy spider from a .bat file

when I run the .bat file which has this text:

@echo off
REM activate Python venv
CALL "D:\python\scrapy_projects\venv\Scripts\activate.bat"
CD "D:\python\scrapy_projects\digikalasellerdata\digikalasellerdata\spiders"
CALL "D:\python\scrapy_projects\venv\Scripts\python.exe" "D:\python\scrapy_projects\venv\Lib\site-packages\scrapy\cmdline.py" crawl my_deactivated -O kobs.csv
pause

I get this error:

Traceback (most recent call last):
  File "D:\python\scrapy_projects\venv\Lib\site-packages\scrapy\cmdline.py", line 8, in <module>
    import scrapy
  File "D:\python\scrapy_projects\venv\lib\site-packages\scrapy\__init__.py", line 12, in <module>
    from scrapy.spiders import Spider
  File "D:\python\scrapy_projects\venv\lib\site-packages\scrapy\spiders\__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "D:\python\scrapy_projects\venv\lib\site-packages\scrapy\http\__init__.py", line 8, in <module>
    from scrapy.http.headers import Headers
  File "D:\python\scrapy_projects\venv\lib\site-packages\scrapy\http\headers.py", line 3, in <module>
    from scrapy.utils.python import to_unicode
  File "D:\python\scrapy_projects\venv\lib\site-packages\scrapy\utils\python.py", line 16, in <module>
    from scrapy.utils.decorators import deprecated
  File "D:\python\scrapy_projects\venv\lib\site-packages\scrapy\utils\decorators.py", line 4, in <module>
    from twisted.internet import defer, threads
  File "D:\python\scrapy_projects\venv\lib\site-packages\twisted\internet\defer.py", line 44, in <module>
    from twisted.internet.interfaces import IDelayedCall, IReactorTime
  File "D:\python\scrapy_projects\venv\lib\site-packages\twisted\internet\interfaces.py", line 26, in <module>
    from twisted.python.failure import Failure
  File "D:\python\scrapy_projects\venv\lib\site-packages\twisted\python\failure.py", line 26, in <module>
    from twisted.python import reflect
  File "D:\python\scrapy_projects\venv\lib\site-packages\twisted\python\reflect.py", line 22, in <module>
    from twisted.python.compat import nativeString
  File "D:\python\scrapy_projects\venv\lib\site-packages\twisted\python\compat.py", line 35, in <module>
    from http import cookiejar as cookielib
  File "D:\python\scrapy_projects\venv\Lib\site-packages\scrapy\http\__init__.py", line 8, in <module>
    from scrapy.http.headers import Headers
ImportError: cannot import name 'Headers' from partially initialized module 'scrapy.http.headers' (most likely due to a circular import) (D:\python\scrapy_projects\venv\lib\site-packages\scrapy\http\headers.py)

any solutions?

Thanks a lot

1 Answers

Instead of directly calling the Python file, use this:

python -m scrapy crawl my_deactivated -O kobs.csv

That's what you activated the virtual environment for.

Related