Why are my tweet search requests only retrieving 200 tweets every time?

user_10404120 0
Viewed 655

I have a script which returns tweets based on a keyword query and then appends this to a CSV. I can't see why in my script, this is only returning 200 tweets every time I run it. It is not the count parameter, because as far as I am aware this returns the number of tweets to return per page, up to a maximum of 100.

Can anyone see what is occurring?

def twitter_search(twitter_api, q, max_results = 3000, **kw):
    search_results = twitter_api.search.tweets(q=q, count = 100, **kw, lang = 'en', tweet_mode='extended', )
    
    statuses = search_results['statuses']
    
    #Iterate through batches of results until we get the number we want
    #Enforce a reasonable limit
    
    max_results = min(5000, max_results)
    
    for _ in range(100):
        try:
            next_results = search_results['search_metadata']['next_results']
        except KeyError as e: #no more results when next_results doesn't exist
            break
    
    #create a dictionary from next_results
        kwargs = dict([kv.split('=') for kv in next_results[1:].split("&")])
    
        search_results = twitter_api.search.tweets(**kwargs)
        statuses += search_results['statuses']
    
        if len(statuses) > max_results:
            break
    
    return statuses

I think it is to do with the cursor iterating over the next batch of results but I do not know why this is happening...

pandas twitter python
4 Answers

Api's only returning a few requests per user. That might be the reason you are not getting all of your tweets.

You can read up about this here Twitter-api (See the Resource information part)

You can also try and read up on Proxycrawler API to get more tweets than you are getting currently.

user_145518420
user_156204110

Twitter API had limits per calls for a given period. You can design function in a way that you can wait on limits reach and move from page to page with cursor.

import tweepy

auth = tweepy.OAuthHandler(API_KEY, API_KEY_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

twitter_api = tweepy.API(auth)

def twitter_search(twitter_api, q, pages = 50 , **kwargs):

    return [page for page in tweepy.Cursor(twitter_api.search,
                                         q=q,
                                         wait_on_rate_limit=True, 
                                         **kwargs
            ).pages(pages)]

This will use cursor to get next pages and wait on rate limit.

user_68582440
Admin1
Related