How to convert pandas dataframe columns to native python data types?

user_6041010 0
Viewed 6385

I have a dataframe whose columns data types need to be mapped to python native data types.

I want to be able to get a dictionary from numpy and convert each column to it's native type.

for example:

{numpy.object_: object,
 numpy.bool_: bool,
 numpy.string_: str,
 numpy.unicode_: unicode,
 numpy.int64: int,
 numpy.float64: float,
 numpy.complex128: complex}

I tried both astype and pd.to_numeric, neither downcasts the column sufficiently.

df['source'] = df['source'].astype(int) returns int32, as does pd.to_numeric

Update:

Most of the comments question the wisdom for doing this. networkx reads dataframes and accepts np datatypes. However the graph cannot be written using json_dumps because of this well documented error: TypeError: Object of type 'int64' is not JSON serializable

Thanks

pandas dataframe python-3.x numpy python
3 Answers

"Native Python type" to pandas (or to numpy) is an object. That's the extent of it. Pandas only knows it's a Python object and act accordingly. Other than that, you cannot have columns of type string, unicode, integers etc.

You can have object columns and store whatever you want inside them, though. Pandas will handle most of the conversion for you at this stage.

df = pd.DataFrame({'A': [1, 2], 
                   'B': [1., 2.], 
                   'C': [1 + 2j, 3 + 4j], 
                   'D': [True, False], 
                   'E': ['a', 'b'], 
                   'F': [b'a', b'b']})

df.dtypes
Out[71]: 
A         int64
B       float64
C    complex128
D          bool
E        object
F        object
dtype: object

for col in df:
    print(type(df.loc[0, col]))

<class 'numpy.int64'>
<class 'numpy.float64'>
<class 'numpy.complex128'>
<class 'numpy.bool_'>
<class 'str'>
<class 'bytes'>

df = df.astype('object')

for col in df:
    print(type(df.loc[0, col]))

<class 'int'>
<class 'float'>
<class 'complex'>
<class 'bool'>
<class 'str'>
<class 'bytes'>
user_22852360

Pandas and by extension Dataframes are built on numpy so you do not get to choose they specific type of type which is stored. Your best bet is to use to_dict and then use that as a poor-mans dataframe. Why would you want to do this?

user_36471670
user_11519100
Related