1-3 'SyntaxError: Unexpected end of JSON input' errors occurring per minute when streaming tweets in Node

Viewed 102

Using express and Node.js, I'm using the twitter streaming API and the needle npm package for accessing APIs to pull tweets related to keywords. The streaming is functional and I am successfully pulling tweets using the following (simplified) code:

const needle = require('needle');
const TOKEN = // My Token 
const streamURL = 'https://api.twitter.com/2/tweets/search/stream';

function streamTweets() {

    const stream = needle.get(streamURL, {
        headers: {
            Authorization: `Bearer ${TOKEN}`
        }
    });

    stream.on('data', (data) => {
        try {
            const json = JSON.parse(data); // This line appears to be causing my error
            const text = json.data.text;
        } catch (error) {
             console.log("error");
        }
    });
}

However, no matter which search term I use (and the subsequent large or small volume of tweets coming through), the catch block will consistently log 1-3 errors per minute, which look like this:

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at PassThrough.<anonymous> (C:\Users\danie\OneDrive\Documents\Personal-Projects\twitter-program\server.js:56:31)
    at PassThrough.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at PassThrough.Readable.push (internal/streams/readable.js:223:10)
    at PassThrough.Transform.push (internal/streams/transform.js:166:32)
    at PassThrough.afterTransform (internal/streams/transform.js:101:10)
    at PassThrough._transform (internal/streams/passthrough.js:46:3)
    at PassThrough.Transform._read (internal/streams/transform.js:205:10).

I've seen previous advice which says that data can be fired in multiple chunks, and to push the chunks to an array i.e. something like the following:

let chunks = [];
        stream.on('data', (dataChunk) => {
chunks.push(dataChunk);
        }).on('end',() => {
// combine chunks to create JSON object
})

But this didn't work either (may have been my implementation but I don't think so) and now I'm wondering if it's perhaps an error with the twitter API, because most of the tweet objects do come through correctly. I should note that the streamTweets() function above is called from an async function, and I am also wondering if that is having something to do with it.

Has anyone else encountered this error? Or does anyone have any idea how I might be fix it? Ideally i'd like 100% of the tweets to stream correctly.

Thanks in advance!

0 Answers
Related