I have a time series data of about 4000 patients that has missing values and I want to impute NaN values using MissForest algorithm in Python on each patient file separately.
The data looks like this:
| HR | Resp | P_ID |
|---|---|---|
| 72.0 | 18.0 | 1 |
| NaN | 15.0 | 1 |
| 80.0 | NaN | 1 |
| NaN | 16.0 | 1 |
| 79.5 | NaN | 1 |
| NaN | 19.0 | 2 |
| 79.5 | 22.5 | 2 |
| NaN | NaN | 2 |
| NaN | 16.0 | 2 |
| 85.0 | NaN | 3 |
| NaN | 14.5 | 3 |
| 76.4 | NaN | 3 |
| NaN | NaN | 4 |
| 80.5 | 19.5 | 4 |
| 75.3 | 18.0 | 4 |
| NaN | 21.5 | 4 |
Now, I want to impute NaN values within the patients data in column based on P_ID. Like it will impute P_ID = 1, then P_ID = 2 and so on. Not the imputation on the whole column. The code I am using will impute NaN on whole column of all patients, not in individual Patients column, then the next patient.
imputer = MissForest(max_iter=12, n_jobs=-1)
X_imputed = imputer.fit_transform(df)
df1 = pd.DataFrame(X_imputed)
df1.head()
I did the Mean Imputation within patient itself using the following code, but can't figure out how I can use it for MissForest.
for i in ['HR','Resp']:
df[i] = df[i].fillna(df.groupby('P_ID')[i].transform('mean'))
One solution is I make 4000 data frames of each patient, impute them using MissForest, then combine them together. That will be a hectic task. So I want a solution with looping over the entire dataframe. Kindly help. Thanks.
