Mismatching Tweets with User Information using Tweepy Paginator

user_14001350 0
Viewed 71

I am trying to match Tweet data with User information using the Paginator, but am encountering problems with missing User information.

This is my code for matching the tweet strings with user metadata:

client = tweepy.Client(bearer_token=config.BEARER_TOKEN, consumer_key=
config.CONSUMER_KEY,consumer_secret= config.CONSUMER_SECRET,access_token=
config.ACCESS_TOKEN,access_token_secret= config.ACCESS_TOKEN_SECRET)

for tweet_batch in tweepy.Paginator(client.search_all_tweets, query=s,
                                    tweet_fields=['context_annotations','created_at', 'public_metrics','author_id', 'lang', 'geo', 'entities'], 
                                    user_fields=['name','username','location','verified','description'],
                                    max_results=100, expansions='author_id'):

    tweet_data = tweet_batch.data
    users = tweet_batch.includes['users']
    print(len(tweet_data),len(users))

The output of the lists were of different lengths:

40  38

tweet_batch consists of the Response form (Personal data removed):

Response(data=[<Tweet id = 123, text = "abc">, <Tweet id = 456, text = "def">], includes={'users':[<User id = 9765, name="foo">]})

The problem is that due to the seperation of 'users' from 'data', there is no method to match corresponding Tweet text to User id if the lengths are mismatched.

Are there any better workarounds?

tweepy twitter python
1 Answers

The advantage of the includes dictionnary is precisely to avoid redundancy: even if several tweets have the same author, the corresponding User object will be sent only once.

You can find the User object thanks to the author_id field in each tweet:

for response in tweepy.Paginator(client.search_all_tweets, [...]):

    tweets = response.data
    users = response.includes["users"]

    # Creating a dictionnary here is a suggestion of the Tweepy documentation

    users = {user["id"]: user for user in users}

    # You can now iterate throw the tweets and associate their User object

    for tweet in tweets:
        author = users[tweet.author_id]
        print(f"The tweet {tweet.id} was written by {author.username}.")
user_162822970
Related