Problem definition

Dear community, I need your help in implementing an LSTM neural network for a classification problem of panel data using Keras. The panel data I am manipulating consists of ids (let's call it id), a timestep for each id (t), n time varying covariates and a binary outcome y. Each id contains a number of timesteps and for each timestep I have my covariates and a unique outcome (0 or 1). I have reason to believe that each covariate for each id can have a certain degree of autocorrelation and henceforth can be considered a small timeseries of t steps. For simplicity, I consider that each id has a fixed number of t observations) with t not a big number (about 10 or so).

Data

Below is a toy example of what the data might look like in my case. In this example, the parameters are 2 individuals, 4 timesteps each, 4 covariates and each observation has a unique binary outcome. Covariates may be considered as (short) timeseries since they might be autocorrelated.

print(df)

[out]:

            A          B            C           D        y
id   t                  
id1  1   1.054127   0.346052    1.091299    -0.058137   0.0
     2   0.621390   -0.204682   -1.056786   0.864572    0.0
     3   1.275124   2.473959    0.264029    -1.047810   0.0 
     4   -0.328441  -0.135891   0.148498    0.470876    1.0
id2  1   0.362969   0.777082    0.197423    -2.225296   0.0
     2   0.227134   0.086731    0.550267    -0.361482   0.0 
     3   0.223526   0.556242    -0.160042   0.675871    1.0
     4   0.070125   0.156659    -2.922709   -1.143887   1.0

I have reason to assume that, for id1, the target at timestep 4 is conditional on the three previous timesteps for that same individual (id1). In addition, The target variable y may contain more than one value of 1 for each individual (as outlined in the case of id2 above). I do not have reason to believe that the data from an individual would affect the result of another (as with many behavior analysis scenarios since every individual is unique).

Prediction problem

What I would like to do is to predict a single outcome for a new individual for whom I have those 4 rows of observation. In other words, based on the historical data of an individual, I would like to know if said individual is likely to have an outcome 1 or 0. If I understand correctly, this can be achieved using an LSTM (alternatively, an RNN) with some data manipulation.

Things I have tried so far

• To start simple, I have tried aggregating every set of id rows into a single row with a single outcome and applied a typical statistical learning approach such as boosted trees and got a model as good as random.

• I looked into shaping it as a survival analysis problem, in vain. I would not be interested in any estimation of a survival function unlike tutorials on how to handle panel data in the medical field (nor would I have access to such data).

• I have tried reshaping my data such that the input is a 3D array in the form of [observations, timesteps, features] where observations are unique ids for an LSTM like so in python :

# separate into features and target
df_feat = df.drop("y", axis = 1)
df_target = df[["y"]]

# get reshaped values for 3D tensor
n_samples = len(df_feat.index.get_level_values('id').unique().tolist())
n_timesteps = 4
n_features = df_feat.shape[1]

# reshape input array to be 3D
X_3D = df_feat.to_numpy().reshape(n_samples, n_timesteps, n_features)
print(X_3D.shape)

[out]:
(2, 4, 4)

However, at this point I get confused as to what my learning instances for the LSTM are and what the outcome y should be shaped like. I have tried having a shape like one outcome per training instance by taking only the last observation for each id (so y=[1,1] and y.shape = (2,) in the toy example above) which technically makes an LSTM script run... but does not capture prior information. Below is the code for such LSTM:

def train_lstm(X_train, y_train, X_valid, y_valid, save_name='best_lstm.h5'):

    # starts a sequential model
    model = Sequential()
    
    # add first lstm hidden layer with 64 units and default keras params
    model.add(LSTM(64, input_shape = (X_train.shape[1], X_train.shape[2]), return_sequences=True))
    
    # add a second hidden lstm layer with 128 units and default keras params
    model.add(LSTM(128, return_sequences = True))
    
    # add one last hidden layer
    model.add(LSTM(64))
    
    # add one dense layer with 2 units and a sigmoid activation function
    model.add(Dense(2, activation = 'sigmoid'))
    
    # define adam optimiser with learning rate
    opt = tf.keras.optimizers.Adam(learning_rate = 0.01)
    
    # compile model with binary cross entropy as loss function and accuracy as metrics
    model.compile(optimizer = opt, loss = 'binary_crossentropy',  metrics = ['accuracy'])
    
    # define early stopping and best model checkpoint parameters
    es = EarlyStopping(monitor = 'val_loss', mode = 'min', verbose = 0, patience = 20)
    mc = ModelCheckpoint(save_name, monitor = 'val_accuracy', mode = 'max', verbose = 0, save_best_only = True)
    
    # train the model using fit method (target vector is one-hot encoded as required by keras)
    history = model.fit(X_train, tf.one_hot(y_train, depth = 2),
                        validation_data = (X_valid, tf.one_hot(y_valid, depth = 2)),
                        epochs = 100, callbacks = [es, mc])
    
    return history

It runs and it makes predictions the way I want them to (for one id of previous history, we can predict one outcome) but results in poor performance since it fails to capture outcomes prior to the last.

• I have carefully read and followed this nicely written medium article by Alexander Laskorunsky which remotely resembles what I am trying to do, and slides the window of K-length frames to capture the prior outcomes (and not just the last as I have done which makes more sense). However, in Alexander's case, he does not consider panel data but rather a multivariate timeseries classification that uses n_timesteps to predict the target using all predictors and all rows even if it overlaps (so not using panel data).

Questions

• Am I right to believe that I need a many to one LSTM architecture?
• How may I divide and reshape training and testing samples such that a new, previously unseen individual which would not be related in any way to other ids can be classified?
• Should each id be considered as one sample / training instance? Should each id be split into training and testing sets and concatenate all training and testing sets to feed to an LSTM architecture?
• Would you be so kind as to provide code snippets on how to correctly split and reshape my data as well as a simple LSTM architecture using keras (or maybe modify my own function above in case I coded it wrong)? No need for basic preprocessing and encoding variables.

Any help or advice / tutorials / articles regarding what architecture is most suitable for that kind of problem is greatly appreciated and thank you in advance for your help!