.tolist() converting pandas dataframe values to bytes

arv24_ 0
Viewed 43

I read a csv file in as a pandas dataframe but now need to use the data to do some calculations.

import pandas as pd

### LOAD DAQ FILE
columns = ["date","time","ch104","alarm104","ch114","alarm114","ch115","alarm115","ch116","alarm116","ch117","alarm117","ch118","alarm118"]
df = pd.read_csv(cal_file,sep='[, ]',encoding='UTF-16 LE',names=columns,header=15,on_bad_lines='skip',engine='python')

### DEFINE SIGNAL CHANNELS
ch104 = df.ch104.dropna() #gets rid of None value at the end of the column
print(ch104)

When I print a ch104 I get the following.

But I cannot do math on it currently as it is a pandas.Series or a string. The datatype is not correct yet. The error if I do calculations is this:

can't multiply sequence by non-int of type 'float'

So what I tried to do is use .tolist() or even list() on the data, but then ch104 looks like this.

I believe the values are now being written as bytes then stores as a list of strings.

Does anyone know how I can get around this or fix this issue? It may be because the original file is UTF-16 LE encoded, but I cannot change this and need to use the files as is.

I need the values for simple calculations later on but they would need to be a float or a double for that. What can I do?

csv list dataframe pandas python
1 Answers

You probably get this error because you're trying to make calculations on some columns considered by pandas as non numeric. The values (numbers in your sense) are for some reason interpreted as strings (in pandas sense).

To fix that, you can change the type of those columns by using pandas.to_numeric :

df_obj = df.select_dtypes(['object'])
df[df_obj.columns] = df_obj.apply(lambda x: x.str.strip()) # to get rid of the extra whitespace

import re
cols = list(filter(re.compile('^(ch|alarm)').match, df.columns.to_list()))

df[cols] = df[cols].apply(pd.to_numeric, errors='coerce')

ch104 = df.ch104.dropna()
#do here your calculations on ch104 serie

Note that the 'coerce' argument will put NaN instead of every bad value in your columns.

errors{‘ignore’, ‘raise’, ‘coerce’}, default ‘raise’ :
If ‘coerce’, then invalid parsing will be set as NaN.

abokey0
Related