Why does pd.concat change the resulting type from int to object?

Admin 1
Viewed 2428

I'm parsing several csv files with Pandas and the concatenating them into one big dataframe. Then, I want to groupby and calculate the mean().

Here's a sample dataframe:

df1.head()

   Time  Node  Packets
0     1     0        0
2     1     1        0
4     1     2        0
6     1     3        0
8     1     4        0

df1.info(verbose=True)

<class 'pandas.core.frame.DataFrame'>
Int64Index: 27972 entries, 0 to 55942
Data columns (total 3 columns):
Time       27972 non-null int64
Node       27972 non-null int64
Packets    27972 non-null int64
dtypes: int64(3)
memory usage: 874.1 KB
None

Then I concatenate them (for the sake of simplicity, three dataframes)

df_total = pd.concat([df1, df2, df3])

df_total.info(verbose=True) results in 

<class 'pandas.core.frame.DataFrame'>
Int64Index: 83916 entries, 0 to 55942
Data columns (total 3 columns):
Time       83916 non-null object
Node       83916 non-null object
Packets    83916 non-null object
dtypes: object(3)
memory usage: 2.6+ MB
None

Finally, I try :

df_total = df_total.groupby(['Time'])['Packets'].mean()

and that's where the error pandas.core.base.DataError: No numeric types to aggregate appears.

While I understand from other posts such as this that Pandas changes the dtype because of the non-null, I couldn't solve my issue with the proposed solutions.

How do I fix this?

pandas concat dataframe python types
2 Answers

I found another post that mentions that dataframes have to be initialized with a dtype, otherwise they are of type object

Did you initialize an empty DataFrame first and then filled it? If so that's probably
why it changed with the new version as before 0.9 empty DataFrames were initialized 
to float type but now they are of object type. If so you can change the 
initialization to DataFrame(dtype=float).

So I added df_total = pd.DataFrame(columns=['Time', 'Node', 'Packets'], dtype=int) to my code and it worked.

Admin1 
 df_total.info(verbose=True)

Your this statement gives info as object, so there is problem in concatenation every value is not int and so mean of object is not possible.

user_46838990
Related