Getting NaNs in X_train and X_test after training/splitting data

user_7278982 0
Viewed 2631

Hello programmers around the world. I have a problem with feeding data into my machine learning model.

I've tried reading a CSV file into python using pandas and then perform the split into training and testing data. After that I scale the results with a StandardScaler and when I get to the feeding part I have NaNs in my training data for some reason. PS: Pretty sure it's because I am missing data but rather because I have infinite data

This what I have in terms of code....

# Importing and organizing required packages and libraries
import pandas as pd;
import numpy as np;
from sklearn.model_selection import train_test_split;
from sklearn.metrics import confusion_matrix, classification_report;
from sklearn.preprocessing import StandardScaler;
from sklearn.ensemble import RandomForestClassifier;
from sklearn.neural_network import MLPClassifier;

#Reading in all of the excel files created from preprocessing.py
dataframe2 = pd.read_csv('dataframe2.csv');
dataframe3 = pd.read_csv('dataframe3.csv');
dataframe4 = pd.read_csv('dataframe4.csv');
dataframe5 = pd.read_csv('dataframe5.csv');

#Function used for creating class labels
def labelCreation(dataframe):
    labels = [];
    index = dataframe['LoC'].index.values;
    for i in range(len(index)):
        if str(dataframe.iloc[i]['Unnamed: 0']) == str(dataframe.iloc[i]['Replacing_line_number']):
            labels.append('1');
        else:
            labels.append('0');
    return labels;

#Picking features for training
def features(dataframe):
    X = dataframe[['Similar_Chars','Similar_Tokens','Edit_Distance','LoC_SemiColon','Replacement_Line_SemiColon','LoC_Open_Bracket_Char',
       'Replacement_Line_Open_Bracket_Char','LoC_Close_Bracket_Char','Replacement_Line_Close_Bracket_Char']];
    return X;

#Training and splitting the data
X_train, X_test, Y_train, Y_test = train_test_split(features(dataframe = dataframe2), labelCreation(dataframe = dataframe2), test_size=0.2);
#X_train, X_test, Y_train, Y_test = train_test_split(features(dataframe = dataframe3), labelCreation(dataframe = dataframe3), test_size=0.2);
#X_train, X_test, Y_train, Y_test = train_test_split(features(dataframe = dataframe4), labelCreation(dataframe = dataframe4), test_size=0.2);
#X_train, X_test, Y_train, Y_test = train_test_split(features(dataframe = dataframe5), labelCreation(dataframe = dataframe5), test_size=0.2);

#Scalling is added in order to get the optimized result
sc = StandardScaler();
X_train = sc.fit_transform(X_train);
X_test = sc.transform(X_test);

#Feeding the data into a random forest classifier model
rfc = RandomForestClassifier(n_estimators = 200);
rfc.fit(X_train, Y_train);
pred_rfc = rfc.predict(X_test);

#Let's see how well the model performed
print(classification_report(Y_test, pred_rfc));
print(confusion_matrix(Y_test, pred_rfc));

#Feeding the data into a neural network model
mlpc=MLPClassifier(hidden_layer_sizes=(11,11,11), max_iter=500);
mlpc.fit(X_train, Y_train);
pred_mlpc = mlpc.predict(X_test);

#Let's see how well the model performed
print(classification_report(Y_test, pred_mlpc));
print(confusion_matrix(Y_test, pred_mlpc));

When I ran all of the code above and then type in X_train[:10] I get this

array([[-0.49869515, -0.39609005, -1.2919533 , -0.96747226,  0.74307391,
     1.02449721,  0.59744363,  1.06693051,  0.58006304],
   [-0.49869515, -0.39609005,  1.22954406,  1.03362137,  0.74307391,
    -0.97608856,  0.59744363, -0.93726817,  0.58006304],
   [        nan,         nan,         nan,         nan,         nan,
            nan,         nan,         nan,         nan],
   [-0.49869515, -0.39609005, -0.67191297, -0.96747226, -1.34576115,
    -0.97608856,  0.59744363, -0.93726817,  0.58006304],
   [        nan,         nan,         nan,         nan,         nan,
            nan,         nan,         nan,         nan],
   [ 0.09153914, -0.39609005, -0.75458501,  1.03362137,  0.74307391,
    -0.97608856,  0.59744363, -0.93726817,  0.58006304],
   [-0.49869515, -0.39609005, -0.50656888, -0.96747226,  0.74307391,
    -0.97608856,  0.59744363, -0.93726817,  0.58006304],
   [-0.49869515, -0.39609005, -0.79592103, -0.96747226,  0.74307391,
     1.02449721, -1.67379807,  1.06693051, -1.72395057],
   [ 0.68177344,  2.20020466,  0.48549566, -0.96747226, -1.34576115,
     1.02449721, -1.67379807,  1.06693051, -1.72395057],
   [-0.20357801, -0.39609005, -0.58924092,  1.03362137,  0.74307391,
     1.02449721,  0.59744363,  1.06693051,  0.58006304]])

Also when I run X_test[:10]I get a similar result

array([[ 3.04271061,  1.33477309, -2.11867374,  1.03362137,  0.74307391,
     1.02449721,  0.59744363,  1.06693051,  0.58006304],
   [-0.49869515,  0.46934152, -0.13454468, -0.96747226, -1.34576115,
     1.02449721,  0.59744363, -0.93726817,  0.58006304],
   [ 0.09153914, -0.39609005, -0.75458501,  1.03362137,  0.74307391,
     1.02449721,  0.59744363,  1.06693051,  0.58006304],
   [-0.20357801, -0.39609005,  1.43622417,  1.03362137, -1.34576115,
     1.02449721,  0.59744363,  1.06693051,  0.58006304],
   [        nan,         nan,         nan,         nan,         nan,
            nan,         nan,         nan,         nan],
   [-0.49869515, -0.39609005, -1.45729739, -0.96747226, -1.34576115,
    -0.97608856,  0.59744363, -0.93726817,  0.58006304],
   [ 1.27200773,  2.20020466, -0.25855274,  1.03362137,  0.74307391,
     1.02449721,  0.59744363,  1.06693051,  0.58006304],
   [-0.20357801, -0.39609005, -1.12660921,  1.03362137, -1.34576115,
    -0.97608856,  0.59744363, -0.93726817,  0.58006304],
   [        nan,         nan,         nan,         nan,         nan,
            nan,         nan,         nan,         nan],
   [-0.49869515, -0.39609005, -0.96126512, -0.96747226, -1.34576115,
    -0.97608856,  0.59744363, -0.93726817,  0.58006304]])

Point being I have no idea why these NaNs are there except for my speculation that I may have infinite values as I made sure I don't have any missing values.

Hope this provides enough background to my problem. If anybody could lend a hand it would be greatly appreciated.

training-data machine-learning nan python-3.x
1 Answers

I had a similar problem myself with having NaN's inside my DataFrame after reading it in from a csv file. My problem was the information being written to the csv file contained the NaN's, which gave this same problem. One option you have is to just search inside your csv file for NaN and see if that is where your problem is. Regardless, if you still wish to pass your data through a neural network without errors, you can just remove them from the data set. I loaded mine in using numpy:

dataset =  np.loadtxt("./CSV Files/Dataset.csv", delimiter=",")
dataset = dataset[~np.any(np.isnan(dataset), axis=1)]

The second line searches through the list of elements in the original array and concatenates it to remove any elements containing NaN, this way the data can be fed through the neural network. My data set was a 2d array, so it would remove the entire array element if it contained a NaN element. One reminder is if you have ground truths in a separate file and they are associated with NaN elements, you are going to want to remove those as well. All you have to do is get the indexes from the data set and remove the elements at those indexes within your ground truth list:

nanIndex = np.argwhere(np.isnan(dataset))
nanIndex = np.delete(nanIndex, 1, 1)
nanIndex = np.unique(nanIndex)
truthValues = np.delete(truthValues, nanIndex)

where truthValues is your 2d list of labels (again this is for a 2d list problem, it is different if it is just 1d). What this code does is create a 2d array of positions where NaN exists in your data set. I simply concatenate it down to just the x values or unique rows. An example of this is where nanIndex originally is: (line 1)

 [[153   0]
 [153   1]
 [153   2]
 [154   0]
 [154   1]]

and is converted to: (line 2)

[[153]
 [153]
 [153]
 [154]
 [154]]

which finally becomes: (line 3)

[[153]
 [154]]

These positions are then just deleted from the ground truth array in line 4.

I hope this helps you overcome your problem, I know it doesn't give you an explicit answer as to why there are NaN's in your data frame, but it can help you avoid the problem of not being able to pass it through your neural network. This is probably not the most efficient manner of getting rid of NaN's in a 2d array, but it works, so if anyone has a better way of doing it, please feel free to inform me!

user_102370520
Related