I"m trying to scrape 365 days of data points from today (2019-12-09 to 2020-03-18). I can't seem to scrape data from somewhere around the dates 2020-02-27 to 2020-03-05. I've tried different keywords, same thing.
I tried splitting the query into two parts: One from 2019-12-09 to 2020-02-27 and Second from 2020-02-27 to 2020-03-18
That successfully made the query run (no code 500), but returned an empty dataframe for the problem time range (2020-02-27 to 2020-03-05).
The worst part is, when I scraped last friday (2020-03-13) for 365 days (which included the problematic data range), it worked fine - no empty dataframe. I have the latest pandas and pytrends.
PS: I can attach the whole python file if you guys want, just felt like it might be unnecessary.
Please help. I have no idea how and why this is happening. My code is as follows:
def get_trends(keyword, days):
i = 0
waiting_time = 60
"""Specify start and end date as well es the required keyword for your query"""
print("Trend keyword: ", type(keyword))
"""Calculating start date from end date, based on given interval"""
end_date = datetime.datetime.date(datetime.datetime.now())
# end_date = end_date - datetime.timedelta(days=20)
print("Today's date: ", end_date)
start_date = end_date - datetime.timedelta(days=days)
print(start_date, end_date)
"""Since we want weekly data for our query, we will create lists which include
the weekly start and end date."""
weekly_date_list = []
# Adds the start date as first entry in our weekly_date_list
start_date_temp = start_date
weekly_date_list.append(start_date_temp)
# This will return in list of weekly datetime.date objects - except the end date
while start_date_temp + datetime.timedelta(days=7) <= end_date:
start_date_temp += datetime.timedelta(days=7)
weekly_date_list.append(start_date_temp)
# This will add the end date to the weekly_date list. We now have a complete list in the specified time frame
if start_date_temp + datetime.timedelta(days=7) > end_date:
weekly_date_list.append(end_date)
print(weekly_date_list)
"""Now we can start to downloading the data via Google Trends API
therefore we have to specify a key which includes the start date
and the end-date with T00 as string for hourly data request"""
"""This list will contain pandas Dataframes of weekly data with the features "date",
"keyword"(which contains weekly scaling between 0 and 100), "isPartial".
Up to this point, the scaling is not correct."""
interest_list = []
# Here we download the data and print the current status of the process
while i < len(weekly_date_list) - 1:
key = str(weekly_date_list[i]) + "T00 " + str(weekly_date_list[i+1]) + "T00"
p = TrendReq()
p.build_payload(kw_list=[keyword], timeframe=key)
interest = p.interest_over_time()
interest_list.append(interest)
print("GoogleTrends Call {} of {} : Timeframe: {} ".format(i + 1, len(weekly_date_list) - 1, key))
i += 1
# print(interest_list)