I need to use the Twitter Stream API to stream tweet data to my firebase cloud function like this:
client.stream('statuses/filter', params, stream => {
stream.on('data', tweet => {
console.log(tweet);
})
stream.on('error', error => {
console.log(error)
})
})
The stream is continuous but the firebase cloud function shuts down after a certain period of time. What solution could I make use of to be able to continuously receive the stream data?
Cloud Functions have a max running time of 540 seconds as documented. You would have to look at probably using a Compute Engine Instance from Google Cloud where you can have code running without limits. Or you could look at using the Google Cloud Scheduler to run your function every x time to get new tweets.
The accepted response suggests running GCE and while it's certainly correct, I'd like to point out that anyone who was interested in Cloud Functions - a serverless solution - might find GAE (App Engine) much more viable for streaming data.
Our application utilises App Engine Standard as an ingestion service and it works like a charm - removing overhead required by GCE. If advanced networking features are required by your app App Engine Flexible or GKE (Kubernetes Engine) might also be something to look at!