Handle PerformanceWarning while creating new columns with map

user_12543122 0
Viewed 1555

Full error:

"PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider using pd.concat instead. To get a de-fragmented frame, use newframe = frame.copy() payouts[x] = ranking[x].map(prizes.set_index('Rank')['Payout'].to_dict())"

lineups = range(1, 5)
prizes = {'Rank':[1, 2, 3], 'Payout':[100, 50, 25]}
prizes = pd.DataFrame(prizes)
payouts = pd.DataFrame(lineups, columns=['Lineup'])

ranking = {'Lineup':[1, 2, 3, 4], 1:[1, 2 , 3, 4], 2:[2, 1, 4, 3], 3:[4, 1, 2, 3], 4:[1, 3, 4, 2]}
ranking = pd.DataFrame(ranking)

for x in range(1, 4):
     payouts[x] = ranking[x].map(prizes.set_index('Rank')['Payout'].to_dict())

payouts = payouts.fillna(-20)
pandas python
1 Answers

Instead of looping, we can create a mapper then apply map to each column in ranking then concat with payouts:

mapper = prizes.set_index('Rank')['Payout'].to_dict()
payouts = pd.concat(
    [payouts,
     ranking[range(1, 5)].apply(lambda s: s.map(mapper)).fillna(-20)],
    axis=1
)

Alternatively we can replace and mask where values are beyond the max prize Rank:

mapper = prizes.set_index('Rank')['Payout'].to_dict()
payouts = pd.concat(
    [payouts,
     ranking[range(1, 5)].replace(mapper)
         .mask(ranking.gt(prizes['Rank'].max()), -20)],
    axis=1
)

Both produce payouts:

   Lineup    1    2    3    4
0       1  100   50  -20  100
1       2   50  100  100   25
2       3   25  -20   50  -20
3       4  -20   25   25   50

*Note in this example ranking contains the necessary information to build the DataFrame without initializing payouts:

mapper = prizes.set_index('Rank')['Payout'].to_dict()
payouts = ranking.copy()  # Create copy of ranking
cols = list(range(1, 5))
payouts[cols] = payouts[cols].apply(lambda s: s.map(mapper)).fillna(-20)

or

mapper = prizes.set_index('Rank')['Payout'].to_dict()
payouts = ranking.copy()  # Create copy of ranking
cols = list(range(1, 5))
payouts[cols] = (
    payouts[cols].replace(mapper).mask(ranking.gt(prizes['Rank'].max()), -20)
)

DataFrame and imports:

import pandas as pd

prizes = pd.DataFrame({'Rank': [1, 2, 3], 'Payout': [100, 50, 25]})
payouts = pd.DataFrame({'Lineup': range(1, 5)})
ranking = pd.DataFrame({
    'Lineup': [1, 2, 3, 4],
    1: [1, 2, 3, 4],
    2: [2, 1, 4, 3],
    3: [4, 1, 2, 3],
    4: [1, 3, 4, 2]
})
user_154978880
Related