fillna method is slower for square bracket column selection than loc for pandas dataframe

user_5147244 0
Viewed 167

I found significant processing time difference in fillna for different column selection techniques of pandas dataframe.

Time taken for fillna of dataframe, whose columns are selected using loc

df1 = df.copy()
t1 = time.time()
df1.loc[:, col] = df1.loc[:, col].fillna(method="ffill")
t2 = time.time()
print(t2-t1)

3.908552885055542

Time taken for fillna of dataframe, whose columns are selected using square bracket:

df1 = df.copy()
t1 = time.time()
df1[col] = df1[col].fillna(method="ffill")
t2 = time.time()
print(t2-t1)

223.85472440719604

This post suggests column selection using loc and square bracket is similar:-
Selecting a list of columns (df[['A', 'B', 'C']] is the same as df.loc[:, ['A', 'B', 'C']] -> selects columns A, B and C)

Can anyone please help why there is time difference? Thanks!!

fillna pandas dataframe numpy python
1 Answers

I have collected few point in 2014, while i was testing it over 2M rows. I found it interesting from SO thread which i collected as follows.

In general, you should use loc for label-based assignment, and iloc for integer/positional based assignment, as the spec guarantees that they always operate on the original.

Better to look at select a subset of a DataFrame

- loc is faster, because it does not try to create a copy of the data.

- loc is meant to modify your existing dataframe inplace, which is more memory efficient.

- loc is predictable, it has one behavior.

- df.loc's syntax is explicit, with df.loc[indexer] you know automatically that df.loc is selecting rows. In contrast, it is not clear if df[indexer] will select rows or columns (or raise ValueError) without knowing details about indexer and df.

When using loc

df.loc[:] = Dataframe

df.loc[int] = Dataframe if you have more than one column and Series if you have only 1 column in the dataframe

df.loc[:, ["col_name"]] = Dataframe if you have more than one row and Series if you have only 1 row in the selection

df.loc[:, "col_name"] = Series

Not using loc

df["col_name"] = Series

df[["col_name"]] = Dataframe

look at here for some interesting detailsPerformance Consideration on multiple columns "Chained Assignment" with and without using .loc

user_57048630
Related