Pandas to risk adjust surgery data - how to pass a list for 20,000 patients?

user_11795964 0
Viewed 141

I am a surgeon trying to use hcuppy to risk adjust surgical operation data. I have a pandas dataframe - 500k records and 56 columns. Calling the adjustment module (elixhauser) expects a list of diagnostic codes and returns a weighted result for risk adjustment related to the diagnostic code.

eg


In[] : from hcuppy.elixhauser import ElixhauserEngine
In[] : ee = ElixhauserEngine()
In[] : out = ee.get_elixhauser(["C711"])

In [] : out
Out[]: {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mrtlt_scr': 7}

So that's good, I get the risk adjustment for having a brain tumour. However I have up to 20 diagnoses for each patient and would like to run the conversion on each diagnosis for cohorts of patients with up to 20,000 patients. At the moment I am limiting it to eg just 4 as below.

df.iloc[0:10, 7:11]

    DIAG_02 DIAG_03 DIAG_04 DIAG_05
0   NaN     NaN     NaN     NaN
1   M7962   NaN     NaN     NaN
2   G800    Q798    NaN     NaN
3   G992-A  M4720D  G551-A  F101-
4   I10X    NaN     NaN     NaN
5   G971    G35X    Z881    N390
6   Z864-   NaN     NaN     NaN
7   F329    NaN     NaN     NaN
8   Z992-   E669-   K219-   I10X-
9   M510    G992    M4806   I10X

if I pass

out = ee.get_elixhauser([df.iloc[0:, 7:11]])

I get

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-78-83647b38b5dc> in <module>
----> 1 out = ee.get_elixhauser([df.iloc[0:, 6:11]])

/opt/anaconda3/lib/python3.7/site-packages/hcuppy/elixhauser.py in get_elixhauser(self, dx_lst)
     93             dx_lst = [dx_lst]
     94 
---> 95         dx_set = {dx.strip().upper().replace(".","") for dx in dx_lst}
     96         rawgrp_lst = [grp for grp in {search(dx) for dx in dx_set}
     97                         if grp not in {"", "NONE"}]

/opt/anaconda3/lib/python3.7/site-packages/hcuppy/elixhauser.py in <setcomp>(.0)
     93             dx_lst = [dx_lst]
     94 
---> 95         dx_set = {dx.strip().upper().replace(".","") for dx in dx_lst}
     96         rawgrp_lst = [grp for grp in {search(dx) for dx in dx_set}
     97                         if grp not in {"", "NONE"}]

/opt/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py in __getattr__(self, name)
   5128             if self._info_axis._can_hold_identifiers_and_holds_name(name):
   5129                 return self[name]
-> 5130             return object.__getattribute__(self, name)
   5131 
   5132     def __setattr__(self, name: str, value) -> None:

AttributeError: 'DataFrame' object has no attribute 'strip'

I was hoping to add the risk adjustment output as separate columns. As you can see I am very much a novice so need simple solutions.

update in light of parfait comments below.

Running

def clean_get_elixhauser(val):
  try:
     result = ee.get_elixhauser(val)
  except Exception as e:
     result = e             # RETURN EXCEPTION MESSAGE - ADJUST AS NEEDED

  return result if pd.isnull(val) else float('nan')

new_df = df.iloc[0:10, 7:11].copy().applymap(clean_get_elixhauser)

on the same dataframe returns,

DIAG_02 DIAG_03 DIAG_04 DIAG_05
0   'float' object has no attribute 'strip' 'float' object has no attribute 'strip' 'float' object has no attribute 'strip' 'float' object has no attribute 'strip'
1   NaN 'float' object has no attribute 'strip' 'float' object has no attribute 'strip' 'float' object has no attribute 'strip'
2   NaN NaN 'float' object has no attribute 'strip' 'float' object has no attribute 'strip'
3   NaN NaN NaN NaN
4   NaN 'float' object has no attribute 'strip' 'float' object has no attribute 'strip' 'float' object has no attribute 'strip'
5   NaN NaN NaN NaN
6   NaN 'float' object has no attribute 'strip' 'float' object has no attribute 'strip' 'float' object has no attribute 'strip'
7   NaN 'float' object has no attribute 'strip' 'float' object has no attribute 'strip' 'float' object has no attribute 'strip'
8   NaN NaN NaN NaN
9   NaN NaN NaN NaN

So, where I have NaN in the original dataframe - 'float' object has no attribute 'strip' is returned and where I have a code I get NaN. I am not sure what is happening here.

pandas data-analysis python
1 Answers

Because ee.get_elixhauser is not a method that receives a whole data frame or array as an input parameter but does receive a scalar value, you will need to run method across every element of the data frame. Therefore, consider DataFrame.applymap assuming all data frame elements are the same type. Also, to avoid issues with assigning by slices due to iloc, use copy() before calling method.

new_df = df.iloc[0:10, 7:11].copy().applymap(ee.get_elixhauser)

Additionally, you may need to handle NaNs and if ee.get_elixhauser raises an exception:

def clean_get_elixhauser(val):
  try:
     result = ee.get_elixhauser([val]) 
  except Exception as e:
     result = e             # RETURN EXCEPTION MESSAGE - ADJUST AS NEEDED

  return result if pd.notnull(val) else float('nan')

new_df = df.iloc[0:10, 7:11].copy().applymap(clean_get_elixhauser)

An additional challenge is that the method returns a dictionary which will be mapped as new elements in returned data frame. (Note: below does not use actual module hcuppy.elixhauser call but the posted result by OP for demonstration to avoid installing a module on my end).

def demo_get_elixhauser(val):
  result = {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mrtlt_scr': 7}

  return result if pd.notnull(val) else float('nan')

new_df = df.applymap(demo_get_elixhauser)

new_df
#                                              DIAG_02  ...                                            DIAG_05
# 0                                                NaN  ...                                                NaN
# 1  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...                                                NaN
# 2  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...                                                NaN
# 3  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...
# 4  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...                                                NaN
# 5  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...
# 6  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...                                                NaN
# 7  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...                                                NaN
# 8  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...
# 9  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...  ...  {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mr...

To fix, you can retrieve one element of returned dictionary by adjusting user defined method

def clean_get_elixhauser(val):
  result = {'cmrbdt_lst': ['TUMOR'], 'rdmsn_scr': 15, 'mrtlt_scr': 7}

  return result['mrtlt_scr'] if pd.notnull(val) else float('nan')

mrtlt_df = df.applymap(clean_get_elixhauser)
  
mrtlt_df
#    DIAG_02  DIAG_03  DIAG_04  DIAG_05
# 0      NaN      NaN      NaN      NaN
# 1      7.0      NaN      NaN      NaN
# 2      7.0      7.0      NaN      NaN
# 3      7.0      7.0      7.0      7.0
# 4      7.0      NaN      NaN      NaN
# 5      7.0      7.0      7.0      7.0
# 6      7.0      NaN      NaN      NaN
# 7      7.0      NaN      NaN      NaN
# 8      7.0      7.0      7.0      7.0
# 9      7.0      7.0      7.0      7.0
user_14224510
Related