I have been trying to do some time series prediction using AWS Sagemaker's DeepAR. DeepAR requires the data as one line JSON inputs. I continue to get error in parsing JSON error and I think its because my "target" has nan values in it. Does anyone know how I could properly preprocess the data or get rid of this nan so that I can get rid of this error? Here is how I preprocessed the initial data:
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os
import pandas as pd
data = pd.read_csv('./data/bitstampUSD_1-min_data_2012-01-01_to_2020-04-22.csv')
data.isnull().values.any()
from datetime import datetime
data.dropna(subset = ["Weighted_Price"], inplace=True)
data.reset_index(drop=True, inplace=True) ##Too many null values so wanted to drop it and reindex
data['date'] = pd.to_datetime(data['Timestamp'],unit='s').dt.date
data.drop(['Timestamp'], axis=1)
daily_price = data['Weighted_Price'].copy()
print(daily_price.shape)
group = data.groupby('date')
Daily_Price_mean = group['Weighted_Price'].mean()
Then I created a function to create a time series:
def make_time_series(mean_power_df, years, freq='D', start_idx=4):
'''Creates as many time series as there are complete years. This code
accounts for the leap year, 2016.
:param Daily_Price_mean: A dataframe of bitcoin weighted price, averaged by day.
This dataframe should also be indexed by a datetime.
:param years: A list of years to make time series out of, ex. ['2013', '2014'].
:param freq: The frequency of data recording (D = daily)
:param start_idx: The starting dataframe index of the first point in the first time series.
The default, 16, points to '2013-01-01'.
:return: A list of pd.Series(), time series data.
'''
# store time series
time_series = []
# store leap year in this dataset
leap = '2012'
# create time series for each year in years
for i in range(len(years)):
year = years[i]
if(year == leap):
end_idx = start_idx+366
else:
end_idx = start_idx+365
# create start and end datetimes
t_start = year + '-01-01' # Jan 1st of each year = t_start
t_end = year + '-12-31' # Dec 31st = t_end
# get global consumption data
data = mean_power_df[start_idx:end_idx]
# create time series for the year
index = pd.date_range(start=t_start, end=t_end, freq=freq)
time_series.append(pd.Series(data=data, index=index))
start_idx = end_idx
# return list of time series
return time_series
full_years = ['2012', '2013', '2014']
freq='D' # daily recordings
# make time series
time_series = make_time_series(Daily_Price_mean, full_years, freq=freq)
def create_training_series(complete_time_series, prediction_length):
'''Given a complete list of time series data, create training time series.
:param complete_time_series: A list of all complete time series.
:param prediction_length: The number of points we want to predict.
:return: A list of training time series.
'''
# get training series
time_series_training = []
for ts in complete_time_series:
# truncate trailing 30 pts
time_series_training.append(ts[:-prediction_length])
return time_series_training
# set prediction length
prediction_length = 30 # 30 days ~ a month
time_series_training = create_training_series(time_series, prediction_length)
I then wrote a function to convert series to json this way:
def series_to_json_obj(ts):
'''Returns a dictionary of values in DeepAR, JSON format.
:param ts: A single time series.
:return: A dictionary of values with "start" and "target" keys.
'''
# get start time and target from the time series, ts
json_obj = {"start": str(ts.index[0]), "target": list(ts)}
return json_obj
ts = time_series[0]
json_obj = series_to_json_obj(ts)
tst = time_series_training[0]
json_objt = series_to_json_obj(tst)
print(json_obj)
print(json_objt)
When I begin to train the data, I start getting Training job failed "Error in parsing JSON" . When I looked at the json_obj output I saw that it was in this form:
{'start': '2012-01-01 00:00:00', 'target': [nan, nan, nan, 5.208159313655556, 6.2841271510761905....]}. I think nan values are causing this to fail but I dont know how to get rid of it especially since I dropped all nan values in the beginning. Can someone please help?