How to convert panda column to int while it has NULL values?

user_5712621 0
Viewed 3997

so I'm working on my CSV file, it has a blank like cell " " after every sentence as shown in the picture below.

when I print the columns type using:

print(data.dtypes)

I get that they are all objects, however I want the columns word_id, head_pred_id, sent_id and run_id to be int64.

when I convert column datatype using:

data.word_id = data.word_id.astype(int)

I get an error : invalid literal for int() with base 10:' '

So I thought the blank spaced-cells are making the problem so I replaced them in CSV file itself with NULL instead.

Now the 4 columns type are automatically set to "Float64" however when I perform something on them I get the same error : ValueError: invalid literal for int() with base 10: ''

I double checked if there's a cell that I missed but I didn't miss any the blank cells are all set to NULL in my CSV file.

Below is a snippet of code where the error appears:

def encode_inputs(sents):
        """
        Given a dataframe which is already split to sentences,
        encode inputs for rnn classification.
        Should return a dictionary of sequences of sample of length maxlen.
        """
        word_inputs = []
        pred_inputs = []
        pos_inputs = []


        assert(all([len(set(sent.run_id.values)) == 1
                    for sent in sents]))


        run_id_to_pred = dict([(int(sent.run_id.values[0]),
                                get_head_pred_word(sent))
                               for sent in sents]) ***ERROR HERE****

and this is the variable "sents" that's sent to the above function


def get_sents_from_df( df):

      #Split a data frame by rows accroding to the sentences
      return [df[df.run_id == run_id]
            for run_id
            in sorted(set(df.run_id.values))]

Snippet of my CSV File

pandas csv excel python
2 Answers

First convert nonnumeric values (like empty strings) to NaNs and then if use pandas 0.24+ is possible convert column to integers:

data.word_id = pd.to_numeric(data.word_id, errors='coerce').astype('Int64')
user_29010020

There is a property of coerce in pd.numeric() function
data['word_id']= pd.to_numeric(data['word_id'], errors='coerce').astype(int)

in case of multiple columns
1. create a list of columns
col =['word_id','head_pred_id']
df[col] = df[col].apply(lambda x :pd.to_numeric(x,errors='coerce').astype(int),axis=0)

user_116079860
Related