I have some data pipeline code that applies transformation / cleanup logic to columns of a Pandas dataframe based on their name.
Right now I'm iterating over the columns using df.iteritems() which according to this guide on optimizing Pandas apply functions is better than crude looping but is "the least efficient way to run most standard functions".
I'd like to improve the performance of this code either by taking advantage of the Pandas's ability to vectorize these operations, or some other parallel approach.
All of the worked examples I have seen illustrate how to do this row-wise (eg, compute on a Series instead of computing on a single row) but I haven't been able to find a good example of how to do this column-wise.
Here is a reproducible / toy example using the Boston dataset from scikit learn. Desired outcome is to implement the cleaning logic in a vectorized / parallel manner (without using .iteritems() or looping). Thanks!
from typing import Callable
# sample df from sklearn
from sklearn import datasets
boston = datasets.load_boston()
boston = pd.DataFrame(boston.data, columns=boston.feature_names)
boston.head()
def double_it(col: pd.Series) -> pd.Series:
return col.multiply(2)
def make_string(col: pd.Series) -> pd.Series:
return col.astype(str)
def do_nothing(col: pd.Series) -> pd.Series:
return col
def match_cleaner(col_name: str) -> Callable:
if col_name in ['ZN', 'NOX', 'INDUS', 'AGE']:
return double_it
elif col_name in ['TAX', 'DIS', 'CHAS', 'PTRATIO']:
return make_string
else:
print(col_name)
return do_nothing
for key, value in boston.iteritems():
cleaning_func = match_cleaner(key)
boston.loc[:, key] = cleaning_func(value)
# confirm changes
boston.head()
print(boston.dtypes)