I'm using snscrape to download a number of tweets. I use a simple script like below to scrape tweets matching a search query.
import snscrape.modules
query = '''(corona OR #covid) until:2020-02-02 since:2020-01-31'''
scraper = snscrape.modules.twitter.TwitterSearchScraper(query=query)
for tweet in scraper.get_items():
print(tweet.content)
break
On my local machine, this runs fine. However, when I deploy to a Heroku instance, I get an error - "unable to find guest token", raised by this method in the source code. I'm having a lot of trouble figuring out what is happening in this part of the code, as it's not documented at all, and the fact that the error doesn't occur locally makes debugging hard. Can anyone help me understand what this method is doing and how I can get it to run on Heroku and not just locally? A stack trace from the heroku cli is below.
2020-10-30T18:55:00.849163+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 131, in get_items
2020-10-30T18:55:00.849284+00:00 app[worker.1]: guestToken = self._get_guest_token()
2020-10-30T18:55:00.849289+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 109, in _get_guest_token
2020-10-30T18:55:00.849496+00:00 app[worker.1]: raise snscrape.base.ScraperException('Unable to find guest token')
2020-10-30T18:55:00.849497+00:00 app[worker.1]: snscrape.base.ScraperException: Unable to find guest token