How to add cell to pd.DataFrame but keep the type (np.uint64) of value?

user_16745187 0
Viewed 115

when I have a DataFrame with np.uint64 values

import pandas as pd
import numpy as np

df = pd.DataFrame(columns=['a','b'])
df.a = [1, 3]
df.b = [1, 3]
df = df.astype({'a': np.uint64, 'b': np.uint64})
print(df.info())

it returns

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   a       2 non-null      uint64
 1   b       2 non-null      uint64
dtypes: uint64(2)
memory usage: 160.0 bytes
None

When I now try to add a cell to the DataFrame:

x = np.uint64(5)
print(type(x))

returns

<class 'numpy.uint64'>

and

df.at[2, 'a'] = x
print(type(df.at[2, 'a']))
print(df.dtypes)

returns

<class 'numpy.float64'>
<class 'pandas.core.frame.DataFrame'>
Int64Index: 3 entries, 0 to 2
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   a       3 non-null      float64
 1   b       2 non-null      float64
dtypes: float64(2)
memory usage: 180.0 bytes
None

Why does the type of the columns and values change from np.uint64 to float64?

pandas dataframe python-3.x numpy types
3 Answers

I believe it is because when you do:

x = np.uint64(5)
df.at[2, 'a'] = x

It generates a NaN for column B and pandas does not allow NaN to be integers which is what uint(64).

If you do add a row that does not have any nulls and convert the dataframe back to unit64 it will allow that:

df = df.astype({'a': np.uint64, 'b': np.uint64})

If that answers your question please mark as a solution!

user_110528010

Note that numpy uint64 cannot be null, which is why it gets auto-converted to float64 after the nan appears.

However pandas does have a nullable integer type in dtype=pd.Int64Dtype() (or aliased as dtype='Int64'):

df = pd.DataFrame({'a': [1, 3], 'b': [1, 3]}, dtype='Int64')

#    a  b
# 0  1  1
# 1  3  3

df.dtypes

# a    Int64
# b    Int64
# dtype: object

df.loc[2, 'a'] = np.uint64(5)

#    a     b
# 0  1     1
# 1  3     3
# 2  5  <NA>

df.dtypes

# a    Int64
# b    Int64
# dtype: object
user_131383640

Update and following question:

My solution to solve the problem:

Instead of using np.uint64 values I cast them beforehand to np.int64.

df = pd.DataFrame(columns=['a','b'])
df.a = [1, 3]
df.b = [1, 3]
df = df.astype({'a': np.int64, 'b': np.int64})
print(df.dtypes)
# a    int64
# b    int64
# dtype: object

In case one I add again one value:

x = np.int64(5)
print(type(x))  # <class 'numpy.int64'>
df.at[2, 'a'] = x
print(type(df.at[2, 'a']))  # <class 'numpy.float64'>
print(df.dtypes)
# a    float64
# b    float64
# dtype: object

Again, both columns are cast to float because NaN does not allow integers. Right?

But in case two, when I do the following instead:

df = pd.DataFrame(columns=['a','b'])
df.a = [1, 3]
df.b = [1, 3]
df = df.astype({'a': np.int64, 'b': np.int64})
print(df.dtypes)
# a    int64
# b    int64
# dtype: object

df_row = pd.DataFrame(columns=['a','b'])
df_row.a = [5]
df_row.b = [5]
print(df_row.dtypes)
# a    int64
# b    int64
# dtype: object

df = df.append(df_row, ignore_index=True)
print(df.dtypes)
# a    int64
# b    int64
# dtype: object

it means that with append in this situation the dataframe does not change the datatype.

But again, if do the same with np.uint64:

df = pd.DataFrame(columns=['a','b'])
df.a = [1, 3]
df.b = [1, 3]
df = df.astype({'a': np.uint64, 'b': np.uint64})
print(df.dtypes)
# a    uint64
# b    uint64
# dtype: object

df_row = pd.DataFrame(columns=['a','b'])
df_row.a = [np.uint64(5)]
df_row.b = [np.uint64(5)]
print(df_row.dtypes)
# a    int64
# b    int64
# dtype: object

df = df.append(df_row, ignore_index=True)
print(df.dtypes)
# a    float64
# b    float64
# dtype: object

The columns types of df_row explains that later at the appendthe types are changed to float64

But why is the datatype in df_row.a = [np.uint64(5)] changed to int64?

user_167451870
Related