Randomly replace 10% of dataframe with NaNs?

user_8768674 0
Viewed 418

I have a randomly generated 10*10 dataset and I need to replace 10% of dataset randomly with NaN.

import pandas as pd
import numpy as np

Dataset = pd.DataFrame(np.random.randint(0, 100, size=(10, 10)))
pandas numpy python
1 Answers

Try the following method. I had used this when I was setting up a hackathon and needed to inject missing data for the competition. -

  1. You can use np.random.choice to create a mask of the same shape as the dataframe. Just make sure to set the percentage of the choice p for True and False values where True represents the values that will be replaced by nans.

  2. Then simply apply the mask using df.mask

import pandas as pd
import numpy as np

p = 0.1 #percentage missing data required

df = pd.DataFrame(np.random.randint(0,100,size=(10,10)))

mask = np.random.choice([True, False], size=df.shape, p=[p,1-p])
new_df = df.mask(mask)
print(new_df)

      0   1     2   3     4     5     6   7   8     9
0  50.0  87   NaN  14  78.0  44.0  19.0  94  28  28.0
1   NaN  58   3.0  75  90.0   NaN  29.0  11  47   NaN
2  91.0  30  98.0  77   3.0  72.0  74.0  42  69  75.0
3  68.0  92  90.0  90   NaN  60.0  74.0  72  58   NaN
4  39.0  51   NaN  81  67.0  43.0  33.0  37  13  40.0
5  73.0   0  59.0  77   NaN   NaN  21.0  74  55  98.0
6  33.0  64   0.0  59  27.0  32.0  17.0   3  31  43.0
7  75.0  56  21.0   9  81.0  92.0  89.0  82  89   NaN
8  53.0  44  49.0  31  76.0  64.0   NaN  23  37   NaN
9  65.0  15  31.0  21  84.0   7.0  24.0   3  76  34.0

EDIT:

Updated my answer for the exact 10% values that you are looking for. It uses itertools and sample to get a set of indexes to mask, and then sets them to nan values. Should be exact as you expected.

from itertools import product
from random import sample

p = 0.1
n = int(df.shape[0]*df.shape[1]*p)          #Calculate count of nans

#Sample exactly n indexes
ids = sample(list(product(range(df.shape[0]), range(df.shape[1]))), n)
idx, idy = list(zip(*ids))

data = df.to_numpy().astype(float)            #Get data as numpy
data[idx, idy]=np.nan                         #Update numpy view with np.nan

#Assign to new dataframe
new_df = pd.DataFrame(data, columns=df.columns, index=df.index)
print(new_df)

      0     1     2     3     4     5     6     7     8     9
0  52.0  50.0  24.0  81.0  10.0   NaN   NaN  75.0  14.0  81.0
1  45.0   3.0  61.0  67.0  93.0   NaN  90.0  34.0  39.0   4.0
2   1.0   NaN   NaN  71.0  57.0  88.0   8.0   9.0  62.0  20.0
3  78.0   3.0  82.0   1.0  75.0  50.0  33.0  66.0  52.0   8.0
4  11.0  46.0  58.0  23.0   NaN  64.0  47.0  27.0   NaN  21.0
5  70.0  35.0  54.0   NaN  70.0  82.0  69.0  94.0  20.0   NaN
6  54.0  84.0  16.0  76.0  77.0  50.0  82.0  31.0   NaN  31.0
7  71.0  79.0  93.0  11.0  46.0  27.0  19.0  84.0  67.0  30.0
8  91.0  85.0  63.0   1.0  91.0  79.0  80.0  14.0  75.0   1.0
9  50.0  34.0   8.0   8.0  10.0  56.0  49.0  45.0  39.0  13.0
user_47559540
Related