Can someone point me in the good direction to understand the (seeming) inconsistency in the way to define dtype during pandas.read_csv ?
dtype = int # --> produces error if blank values
dtype = int32, int64, and Int64 # --> are not defined
dtype = 'Int64' # --> correctly reads csv file as ints and with blank values
- Why are quotes required for 'Int64' but not for str, float, int, object ?
- I did not find a reference that clearly defines the valid list of dtypes for pandas.read_csv. Does this exist somewhere ?
import pandas as pd; print(pd.__version__)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
MY_DTYPES = {
'date_string': str,
'description': str,
# 'ValueError_Integer_column_has_NA_values': int,
# 'int32_is_not_defined': int32,
# 'int64_is_not_defined': int64,
# 'Int_64_is_not_defined': Int64,
'Int64_with_quote_and_NaN': 'Int64', # !! THIS WORKS !!
'quantity': float,
'total': float}
f = 'dataset.csv'
df = pd.read_csv(f, dtype = MY_DTYPES)
df.head(15)
date_string description Int64_with_quote_and_NaN quantity total
0 201202 "Lorem ips 513 186.0 4.0
1 200909 um dolor 601 502.0 13.0
2 201701 sit NaN 462.0 20.0
3 201401 amet, 513 934.0 206.0
4 201202 consectetur 513 NaN 194.0
5 200710 adipiscing 602 570.0 930.0
6 200501 elit, 513 160.0 NaN
7 200808 sed NaN 508.0 461.0
8 201906 do 513 316.0 3.0
9 201009 eiusmod NaN 994.0 1.0
10 NaN NaN 513 709.0 0.0
11 201905 incididunt 513 318.0 6.0
12 201612 ut 513 NaN 1.0
13 201506 labore 513 901.0 74.0
14 201002 NaN 625 33.0 739.0