Incomprehension of input data for the LSTM model

user_10682900 0
Viewed 87

I'm facing a problem I can't solve. Indeed, I try to create a model LSTM with keras, but I don't understand what the input data format should be. My data train and my data test look like this:

date/value/value/value/value/value_i_want_to_predict

I've seen some people doing this:

from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler(feature_range = (0, 1))
training_set_scaled = sc.fit_transform(training_set)X_train = []
y_train = []
for i in range(60, len(training_set_scaled)):
    X_train.append(training_set_scaled[i-60: i, 0])
    y_train.append(training_set_scaled[i, 0])
    X_train, y_train = np.array(X_train), np.array(y_train)

But if I do that how do I predict my features without modifying the test data set?

I have a hard time understanding why we do this. Moreover, what I would like to use the values to predict the target in the last column. With this method I feel like I have to change the format of the data test and it's important that I can test the model on test data that are different and that I don't have to change.

Can someone help me?

EDIT

scaler.fit(df_train_x)

X_train = scaler.fit_transform(df_train_x)
X_test = scaler.transform(df_test_x)   

y_train = np.array(df_train_y)
y_train = np.insert(y_train, 0, 0)
y_train = np.delete(y_train, -1)

The shape of the data is: (2420, 7)

That what I did. But The shape still remain 2D. So i used :

generator = TimeseriesGenerator(X_train, y_train, length=n_input, batch_size=32)

And the input shape of first layer is:

model.add(LSTM(150, activation='relu',  return_sequences=True,input_shape=(2419, 7)))

but when i fit the generator to the model:

ValueError: Error when checking target: expected dense_10 to have 3 dimensions, but got array with shape (1, 1)

i really don't understand

lstm keras python
1 Answers

I'm not sure to fullly understand your question but I will try my best. I think the code you provided is problem specific, meaning it maybe not suitable for your imlementation.

For an LSTM (and for pretty much any neural network) you always want to scale your data before feeding it to the model. This helps avoid having completely different data ranges across your features. The MinMaxScaler scale your features to the range provided. For an explanation of why do you need scaling, you can have a look at this article.

Usualy, you want to first split your dataset in training and testing sets, using for example the train_test_split function of sklearn, then scale your features.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X = data.drop("feature_I_want_to_predict",axis=1)
y = data["feature_I_want_to_predict"]

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

That way, X_train represent your training data, and y_train represent your labels for the training data. (and similarly for the test data) I here used the StandardScaler instead of the MinMaxScaler. The standard scaler substracts the mean of the feature then divides by the standard deviation.

user_88142290
Related