I want to retrieve data based on certain keyword from Twitter API using tweepy to pandas python. I already have elevated developer account that can retrieve up to 2mio data/month per project based on this official documentation. I also already can get the twitter data with this code. But, I want to retrieve more data (at least 100k tweet, regardless it's in 1 time execution or may be up to 10x execution. But, no matter the number I input at the max_result parameter, there are only about 50 data I can get.
So, is anyone know how is the solution? I already try pagination like this tutorial but got an error like this that say if the feature is still on development
auth = tweepy.OAuthHandler('my api key', 'my api key secret')
auth.set_access_token('my access token', 'my access token secret')
api = tweepy.API(auth)
# get tweet based on certain keywords
KEYWORDS = 'flower OR doll'
tweets = api.search(KEYWORDS, result_type="mixed", tweet_mode="extended", max_results = 10000)
columns = ['time', 'user', 'tweet']
data = []
for tweet in tweets:
data.append([tweet.created_at, tweet.user.screen_name, tweet.full_text])
df = pd.DataFrame(data, columns = columns)