This is how I have data which has 1000000 columns. Below is a fraction of it. And I changed values of pred just for people to see easier.There are many users solving cards. about 16000 users and 16000 cards.
user card true pred
0 7933 4169 1.0 0.821682
1 16151 2940 1.0 0.724903
2 15134 2183 1.0 0.826126
3 12775 5977 1.0 0.685955
4 196 14972 1.0 0.719164
5 12994 8631 1.0 0.739311
6 15351 9968 1.0 0.738173
7 5413 7292 1.0 0.685801
8 8232 6662 1.0 0.764592
9 10437 12367 1.0 0.934161
user_pred = data.groupby(by = ['user'], as_index = False).apply(lambda x: x['pred'].sort_values(ascending= False))
And this returns me as below. and I did this as below to get groups in order of pred. This is only the first user data (only user 0) as the group is big.
user index pred
0 5172761 0.934161
2193689 0.826126
544664 0.821682
8472082 0.764592
4678627 0.739311
1742257 0.738173
450349 0.724903
6202140 0.719164
8834527 0.685955
6149264 0.685801
Which says, User 0 has preds and the result is ordered on pred. But what I ultimately want is I want to get this with the associated true column. So the goal would look like this
user index pred true
0 5172761 0.934161 1.0
2193689 0.826126 1.0
544664 0.821682 1.0
8472082 0.764592 1.0
4678627 0.739311 1.0
1742257 0.738173 1.0
450349 0.724903 1.0
6202140 0.719164 1.0
8834527 0.685955 1.0
6149264 0.685801 1.0
Can someone provide a code for this? Thank you in advance. Cheers.