I am crawling Twitter.
But I can not collect all the data.
I am typing the keyword and crawling the url that specified the date.
But there are certainly a number of tweets that can only bring twenty.
(For example, tweets that include the keyword "money" from July 18, 1917 to July 20,)
What is the problem? How should I fix it?
# -*- coding:utf-8 -*-
#python3
import urllib.request
from bs4 import BeautifulSoup
# save txt file
OUTPUT_FILE_NAME = 'output_sample.txt'
# scraping URL
URL = 'https://twitter.com/search?l=&q=money%20since%3A2017-07-18%20until%3A2017-07-20&src=typd&lang=ko'
#crawler
def spider(URL):
source_code_from_url = urllib.request.urlopen(URL)
soup = BeautifulSoup(source_code_from_url, 'lxml', from_encoding='utf-8')
text = ''
for item in soup.find_all('p', {'class': 'tweet-text'}):
text = text + str(item.find_all(text=True)) + "\n"
return text
def main():
open_output_file = open(OUTPUT_FILE_NAME, 'w', encoding='utf-8')
result_text = spider(URL)
open_output_file.write(result_text)
open_output_file.close()
if __name__ == '__main__':
main()