I have many tweets, but only 20 can be brought

Viewed 37

I am crawling Twitter.

But I can not collect all the data.

I am typing the keyword and crawling the url that specified the date.

But there are certainly a number of tweets that can only bring twenty.

(For example, tweets that include the keyword "money" from July 18, 1917 to July 20,)

What is the problem? How should I fix it?

# -*- coding:utf-8 -*-
#python3
import urllib.request
from bs4 import BeautifulSoup


# save txt file
OUTPUT_FILE_NAME = 'output_sample.txt'

# scraping URL
URL = 'https://twitter.com/search?l=&q=money%20since%3A2017-07-18%20until%3A2017-07-20&src=typd&lang=ko'

#crawler
def spider(URL):
    source_code_from_url = urllib.request.urlopen(URL)
    soup = BeautifulSoup(source_code_from_url, 'lxml', from_encoding='utf-8')
    text = ''
    for item in soup.find_all('p', {'class': 'tweet-text'}):

        text = text + str(item.find_all(text=True)) + "\n"
    return text

def main():
    open_output_file = open(OUTPUT_FILE_NAME, 'w', encoding='utf-8')
    result_text = spider(URL)
    open_output_file.write(result_text)
    open_output_file.close()

if __name__ == '__main__':
    main()
0 Answers
Related