How to reshape series to use it in StandardScaler

Viewed 1051

I am trying to perform a time series (LSTM ) on some data that I processed and now I am trying to scale it using the StandScaler from sklearn. Here is my initial preprocessing of data:

import json
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os
import pandas as pd
import sklearn
import torch
import torch.nn as nn
import torch.nn.functional as F

data = pd.read_csv('./data/bitstampUSD_1-min_data_2012-01-01_to_2020-04-22.csv')
data.isnull().values.any()

from datetime import datetime
data.dropna(subset = ["Weighted_Price"], inplace=True)
data.reset_index(drop=True, inplace=True) ##Too many null values so wanted to drop it and reindex


data['date'] = pd.to_datetime(data['Timestamp'],unit='s').dt.date
group = data.groupby('date')
daily_price = group['Weighted_Price'].mean()

daily_price.head()

df_train= daily_price[0:1800] #60%
df_train.shape
df_validation= daily_price[1801:2500] #80%
df_validation.shape
df_test= daily_price[2500:] #60%
df_test.shape

The output of the train, validation and test sets are (1800,) (699,) and (533,) respectively

When I tried to run a standardscaler by doing:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
train_arr = scaler.fit_transform(df_train)
val_arr = scaler.transform(df_validation)
test_arr = scaler.transform(df_test)

I got the following error:

Expected 2D array, got 1D array instead:
array=[  4.47160287   4.80666667   5.         ... 759.70635334 751.50645584
 755.52545612].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample

I tried to redo the previous code the following way:

from sklearn.preprocessing import StandardScaler
df_train_2D = df_train.reshape(1,-1)
df_validation_2D = df_validation.reshape(1,-1)
df_test_2D = df_test.reshape(1,-1)
scaler = StandardScaler()
train_arr = scaler.fit_transform(df_train_2D)
val_arr = scaler.transform(df_validation_2D)
test_arr = scaler.transform(df_test_2D)

But I still get errors: 'Series' object has no attribute 'reshape'

I am not sure how to turn this into an acceptable array for scaling. Can somoene please help and let me know how to solve this? Thanks

1 Answers

DataFrame.reshape() was depreciated in pandas 0.19.

You need to reshape the values of the DataFrame or Series, which you can get by adding .values:

df.values.reshape(1,-1)

Or, in your case:

df_validation_2D = df_validation.values.reshape(1,-1)
Related