I am trying to perform a time series (LSTM ) on some data that I processed and now I am trying to scale it using the StandScaler from sklearn. Here is my initial preprocessing of data:
import json
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os
import pandas as pd
import sklearn
import torch
import torch.nn as nn
import torch.nn.functional as F
data = pd.read_csv('./data/bitstampUSD_1-min_data_2012-01-01_to_2020-04-22.csv')
data.isnull().values.any()
from datetime import datetime
data.dropna(subset = ["Weighted_Price"], inplace=True)
data.reset_index(drop=True, inplace=True) ##Too many null values so wanted to drop it and reindex
data['date'] = pd.to_datetime(data['Timestamp'],unit='s').dt.date
group = data.groupby('date')
daily_price = group['Weighted_Price'].mean()
daily_price.head()
df_train= daily_price[0:1800] #60%
df_train.shape
df_validation= daily_price[1801:2500] #80%
df_validation.shape
df_test= daily_price[2500:] #60%
df_test.shape
The output of the train, validation and test sets are (1800,) (699,) and (533,) respectively
When I tried to run a standardscaler by doing:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
train_arr = scaler.fit_transform(df_train)
val_arr = scaler.transform(df_validation)
test_arr = scaler.transform(df_test)
I got the following error:
Expected 2D array, got 1D array instead:
array=[ 4.47160287 4.80666667 5. ... 759.70635334 751.50645584
755.52545612].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample
I tried to redo the previous code the following way:
from sklearn.preprocessing import StandardScaler
df_train_2D = df_train.reshape(1,-1)
df_validation_2D = df_validation.reshape(1,-1)
df_test_2D = df_test.reshape(1,-1)
scaler = StandardScaler()
train_arr = scaler.fit_transform(df_train_2D)
val_arr = scaler.transform(df_validation_2D)
test_arr = scaler.transform(df_test_2D)
But I still get errors: 'Series' object has no attribute 'reshape'
I am not sure how to turn this into an acceptable array for scaling. Can somoene please help and let me know how to solve this? Thanks