My spider is working both on the local machine and on the Heroku platform, however, I wish to schedule the spider so it works according to the time given through cron expression.
This is my periodic_requests.py file:
from pytz import timezone
import requests, pytz
from apscheduler.schedulers.twisted import TwistedScheduler
from twisted.internet import reactor
def send_request():
requests.post(
url="https://datacrawlapp.herokuapp.com/schedule.json",
data={"project":"datahubsitecrawler",
"spider":"datahubcrawlapp",
"data_url": "https://drive.google.com/file/d/1zBfsWZdoFzoAq6ZELfw-tvvBFcVmCMI7/view?usp=sharing"
}
)
if __name__ == '__main__':
scheduler = TwistedScheduler(timezone=pytz.utc),
scheduler.add_job(send_request, 'cron', day_of_week='mon-sun', hour='19', minute='47')
scheduler.start()
try:
reactor.run() # the script will block here until the crawling is finished
except (KeyboardInterrupt, SystemExit):
pass
Procfile:
web: scrapyd
clock: python periodic_requests
I get error:
File "/app/.heroku/python/lib/python3.8/site-packages/scrapyd/website.py", line 212,
in prep_tab_finished
2022-02-07T08:58:06.697647+00:00 app[web.1]: return '\n'.join(
2022-02-07T08:58:06.697647+00:00 app[web.1]: builtins.TypeError: 'NoneType' object is not iterable:
and then it crashes as shown below:
2022-02-07T08:48:32.411005+00:00 heroku[web.1]: State changed from starting to up
2022-02-07T08:48:32.611649+00:00 heroku[clock.1]: Process exited with status 0
2022-02-07T08:48:32.721748+00:00 heroku[clock.1]: State changed from up to crashed
2022-02-07T08:48:32.742567+00:00 heroku[clock.1]: State changed from crashed to starting
2022-02-07T08:48:41.000000+00:00 app[api]: Build succeeded
2022-02-07T08:48:42.473990+00:00 heroku[clock.1]: Starting process with command `python periodic_requests.py`
2022-02-07T08:48:43.195961+00:00 heroku[clock.1]: State changed from starting to up
2022-02-07T08:48:44.920927+00:00 heroku[clock.1]: Process exited with status 0
2022-02-07T08:48:44.984024+00:00 heroku[clock.1]: State changed from up to crashed
Will appreciate your help!