While the applymap function on DataFrame operates element-wise, the transform function seems to achieve the same thing except claiming to return a like-indexed DataFrame.
Questions:
What's meant by
.transform()returns a like-indexed DataFrame stated in the documentation?
That means .transform() applies a function to every value (or a group, once preceded by groupby) in the DataFrame and returns another DataFrame with the same length as the input, so to emphasize: it keeps the input index labels in the output.
Is there any use case where one of applymap/transform and the other doesn't?
Sure. Here are some examples:
Since applymap performs on all elements of a DataFrame, you can't perform applymap on a Series:
df['Quantity'].transform(lambda x: x+10) # successful
df['Quantity'].apply(lambda x: x+10) # successful
df['Quantity'].applymap(lambda x: x+10) # gives AttributeError: 'Series' object has no attribute 'applymap'
# unless you cast it to DataFrame:
pd.DataFrame(df['Quantity']).applymap(lambda x: x+10) # successful
Another important difference is that despite .applymap() which operates element-wise, .transform() can perform group-wise operations, referred to in the next part.
Moreover, applymap cannot be preceded by groupby.
apply and transform can be interchangeable as long as you perform them on DataFrame column(s). Here is a simple example:
# imagine the following DataFrame
df = pd.DataFrame({'Label': ['A', 'B', 'C', 'A', 'C'],
'Values': [0,1,2,3,4],
'Quantity': [5,6,7,8,9]}, index = list('VWXYZ'))
Label Quantity Values
---------------------------------
V A 5 0
W B 6 1
X C 7 2
Y A 8 3
Z C 9 4
df.loc[:, ['Quantity', 'Values']].apply(lambda x: x+10)
df.loc[:, ['Quantity', 'Values']].transform(lambda x: x+10)
# both of them give the following same result:
Quantity Values
-------------------------
V 15 10
W 16 11
X 17 12
Y 18 13
Z 19 14
The main difference emerges once they follow a groupby operation. For instance:
label_grouping = df.groupby('Label')
label_grouping.apply(lambda x: x.mean())
# output:
Quantity Values
Label
-----------------------
A 6.5 1.5
B 6.0 1.0
C 8.0 3.0
label_grouping.transform(lambda x: x.mean())
# see how `transform` could manage to keeps the input index labels in the output
# output:
Quantity Values
------------------------
V 6.5 1.5
W 6.0 1.0
X 8.0 3.0
Y 6.5 1.5
Z 8.0 3.0
The above example clearly shows how transform can retain input DataFrame indexes; So to get the better of this exclusive feature, the following short example tries to clarify how to benefit from this alignment of the indexes between the input and output of the transform operation, by calculating the percentage of order total that each product represents:
df_sales = pd.DataFrame({'OrderID': [1001,1001,1001,1002,1002],
'Product': ['p1','p2','p3','p1','p4'],
'Quantity': [30,20,70,160,40]})
OrderID Product Quantity
-----------------------------------
0 1001 p1 30
1 1001 p2 20
2 1001 p3 70
3 1002 p1 160
4 1002 p4 40
df_sales['total_per_order'] = df_sales.groupby(['OrderID'])['Quantity'].transform(lambda x: x.sum())
df_sales['pct_of_order'] = df_sales['Quantity'] / df_sales['total_per_order']
OrderID Product Quantity total_per_order pct_of_order
----------------------------------------------------------------------
0 1001 p1 30 120 0.250000
1 1001 p2 20 120 0.166667
2 1001 p3 70 120 0.583333
3 1002 p1 160 200 0.800000
4 1002 p4 40 200 0.200000
It's highly advised to follow this link for a more detailed example: https://pbpython.com/pandas_transform.html
Many aggregation functions are built in directly to the groupby object to save you some typing. specifically some of the common ones to benefit from are (prefixed by gb):
Hope this helped :)
I am new Pandas and was trying to find an answer to the same question. I did find this on the Pandas website (https://pandas.pydata.org/pandas-docs/stable/basics.html)
Since not all functions can be vectorized (accept NumPy arrays and return another array or value), the methods applymap() on DataFrame and analogously map() on Series accept any Python function taking a single value and returning a single value.
I am taking that to mean that functions passed to transform should be vectorized, and the entire series will be passed as an argument to these functions. Functions passed to map (and applymap) do not need to be vectorized, and each element will be passed to the function as map iterates over the series.
> df = pd.DataFrame([['europe', 'france', 68],
> ['europe', 'russia', 144],
> ['asia', 'china', 1398]],
> columns=['continent', 'country', 'population'])
> df
continent country population
0 europe france 68
1 europe russia 144
2 asia china 1398
applymap
> df[['country', 'population']].set_index('country')
.applymap(lambda x: 'big' if x > 100 else 'small')
country
france small
russia big
china big
> df.groupby(['continent'], sort=False)['population'].applymap(max)
AttributeError: 'SeriesGroupBy' object has no attribute 'applymap'
transform
> df[['country', 'population']].set_index('country')
.transform(lambda x: 'big' if x > 100 else 'small')
ValueError: The truth value of a Series is ambiguous.
> df.groupby(['continent'], sort=False)['population'].transform(max)
0 144
1 144
2 1398
Name: population, dtype: int64
> df[df.groupby(['continent'], sort=False)['population'].transform(max)
== df['population']]
continent country population
1 europe russia 144
2 asia china 1398