Error when parsing json during DeepAR training job - How to remove Nan values from target

Viewed 153

I have been trying to do some time series prediction using AWS Sagemaker's DeepAR. DeepAR requires the data as one line JSON inputs. I continue to get error in parsing JSON error and I think its because my "target" has nan values in it. Does anyone know how I could properly preprocess the data or get rid of this nan so that I can get rid of this error? Here is how I preprocessed the initial data:

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os
import pandas as pd


data = pd.read_csv('./data/bitstampUSD_1-min_data_2012-01-01_to_2020-04-22.csv')
data.isnull().values.any()
from datetime import datetime
data.dropna(subset = ["Weighted_Price"], inplace=True)
data.reset_index(drop=True, inplace=True) ##Too many null values so wanted to drop it and reindex


data['date'] = pd.to_datetime(data['Timestamp'],unit='s').dt.date
data.drop(['Timestamp'], axis=1)
daily_price = data['Weighted_Price'].copy()
print(daily_price.shape)
group = data.groupby('date')
Daily_Price_mean = group['Weighted_Price'].mean()

Then I created a function to create a time series:

     def make_time_series(mean_power_df, years, freq='D', start_idx=4):
            '''Creates as many time series as there are complete years. This code
               accounts for the leap year, 2016.
              :param Daily_Price_mean: A dataframe of bitcoin weighted price, averaged by day.
                  This dataframe should also be indexed by a datetime.
              :param years: A list of years to make time series out of, ex. ['2013', '2014'].
              :param freq: The frequency of data recording (D = daily)
              :param start_idx: The starting dataframe index of the first point in the first time series.
                  The default, 16, points to '2013-01-01'. 
              :return: A list of pd.Series(), time series data.
              '''
            
            # store time series
            time_series = []
            
            # store leap year in this dataset
            leap = '2012'
        
            # create time series for each year in years
            for i in range(len(years)):
        
                year = years[i]
                if(year == leap):
                    end_idx = start_idx+366
                else:
                    end_idx = start_idx+365
        
                # create start and end datetimes
                t_start = year + '-01-01' # Jan 1st of each year = t_start
                t_end = year + '-12-31' # Dec 31st = t_end
        
                # get global consumption data
                data = mean_power_df[start_idx:end_idx]
        
                # create time series for the year
                index = pd.date_range(start=t_start, end=t_end, freq=freq)
                
                time_series.append(pd.Series(data=data, index=index))
                
                start_idx = end_idx
            
            # return list of time series
            return time_series
    
    full_years = ['2012', '2013', '2014']
    freq='D' # daily recordings
    
    # make time series
    time_series = make_time_series(Daily_Price_mean, full_years, freq=freq)

def create_training_series(complete_time_series, prediction_length):
    '''Given a complete list of time series data, create training time series.
       :param complete_time_series: A list of all complete time series.
       :param prediction_length: The number of points we want to predict.
       :return: A list of training time series.
       '''
    # get training series
    time_series_training = []
    
    for ts in complete_time_series:
        # truncate trailing 30 pts
        time_series_training.append(ts[:-prediction_length])
        
    return time_series_training

# set prediction length
prediction_length = 30 # 30 days ~ a month

time_series_training = create_training_series(time_series, prediction_length)

I then wrote a function to convert series to json this way:

def series_to_json_obj(ts):
    '''Returns a dictionary of values in DeepAR, JSON format.
       :param ts: A single time series.
       :return: A dictionary of values with "start" and "target" keys.
       '''
    # get start time and target from the time series, ts
    json_obj = {"start": str(ts.index[0]), "target": list(ts)}
    return json_obj

ts = time_series[0]

json_obj = series_to_json_obj(ts)

tst = time_series_training[0]

json_objt = series_to_json_obj(tst)

print(json_obj)
print(json_objt)

When I begin to train the data, I start getting Training job failed "Error in parsing JSON" . When I looked at the json_obj output I saw that it was in this form: {'start': '2012-01-01 00:00:00', 'target': [nan, nan, nan, 5.208159313655556, 6.2841271510761905....]}. I think nan values are causing this to fail but I dont know how to get rid of it especially since I dropped all nan values in the beginning. Can someone please help?

0 Answers
Related