Faster way of sampling pandas rows by group than simple lambda apply

So I have a dataframe df_hist that I'm sampling a row by a group pat_mrn_id. It's pretty darn slow and there's got to be a vectorized way of doing this.

Code example below

import random
import numpy as np
import pandas as pd

N = int(1e8)
A_list = np.random.randint(1, 100, N)
B_list = np.random.randint(1, 100, N)
mrns = [random.randint(0,1000) for i in range(N)]
d = {'pat_mrn_id':mrns,'a_list':A_list,'b_list':B_list}
df_hist = pd.DataFrame(data=d)
df_hist.groupby('pat_mrn_id').apply(lambda x: x.sample(1)).reset_index(drop=True)