My Spider crashes whenever I schedule to production on Heroku

Viewed 25

My spider is working both on the local machine and on the Heroku platform, however, I wish to schedule the spider so it works according to the time given through cron expression.

This is my periodic_requests.py file:

from pytz import timezone
import requests, pytz
from apscheduler.schedulers.twisted import TwistedScheduler
from twisted.internet import reactor


def send_request():
    requests.post(
        url="https://datacrawlapp.herokuapp.com/schedule.json",
        data={"project":"datahubsitecrawler",
        "spider":"datahubcrawlapp",
        "data_url": "https://drive.google.com/file/d/1zBfsWZdoFzoAq6ZELfw-tvvBFcVmCMI7/view?usp=sharing"
    }
    )


    if __name__ == '__main__':
        scheduler = TwistedScheduler(timezone=pytz.utc),
        scheduler.add_job(send_request, 'cron', day_of_week='mon-sun', hour='19', minute='47')
        scheduler.start()

        try:
            reactor.run() # the script will block here until the crawling is finished
        except (KeyboardInterrupt, SystemExit):
            pass

Procfile:

web: scrapyd
clock: python periodic_requests

I get error:

 File "/app/.heroku/python/lib/python3.8/site-packages/scrapyd/website.py", line 212, 
in prep_tab_finished
2022-02-07T08:58:06.697647+00:00 app[web.1]: return '\n'.join(
2022-02-07T08:58:06.697647+00:00 app[web.1]: builtins.TypeError: 'NoneType' object is not iterable:

and then it crashes as shown below:

2022-02-07T08:48:32.411005+00:00 heroku[web.1]: State changed from starting to up
2022-02-07T08:48:32.611649+00:00 heroku[clock.1]: Process exited with status 0
2022-02-07T08:48:32.721748+00:00 heroku[clock.1]: State changed from up to crashed
2022-02-07T08:48:32.742567+00:00 heroku[clock.1]: State changed from crashed to starting
2022-02-07T08:48:41.000000+00:00 app[api]: Build succeeded
2022-02-07T08:48:42.473990+00:00 heroku[clock.1]: Starting process with command `python periodic_requests.py`
2022-02-07T08:48:43.195961+00:00 heroku[clock.1]: State changed from starting to up
2022-02-07T08:48:44.920927+00:00 heroku[clock.1]: Process exited with status 0
2022-02-07T08:48:44.984024+00:00 heroku[clock.1]: State changed from up to crashed

Will appreciate your help!

0 Answers
Related