I have a very large data of 4 million rows, I decided to take the first 25% of the data by this line of code.
df_sample= df.head(int(df.values.shape[0]*0.25))
then I decided to check how many values of the output are lost while sampling. I checked the unique values of both the data frame and the sample by the following code:
sample_outputs=df_sample['output'].nunique()
wholedata_outputs=df['output'].nunique()
I found out that the sample_outputs is ``34, and wholedata_outputs``` is 37, so I have expected that there are 3 values of the output are not existed in the sample. However when I tried this line of code:
list_all=df['output'].unique()
list_sample=df_sample['output'].unique()
main_list = np.setdiff1d(list_all,list_sample)
I expected the length of main_list (the elements that are existing in the whole data and does not exist in the sample) to be 3. However, the length was 7. So, I had to dig more to find what the values that exist in the sample and does not exist in the whole data (I know theoretically speaking it should not happen as I literally just took sample of the whole data). However I found this:
number of unique values of all data is 37
number of unique values of 1st percentile is 34
The elements which don't exist in 1st perecentiles are
0.0005
0.0035
0.0095
0.011
0.014
0.022
0.029
and the values that exist in the sample and not in the whole dataset are:
0.011000000000000001
0.013999999999999999
0.022000000000000002
0.028999999999999998
So I had to print the unique values of the whole data to search about the final 4 values that exist in the sample and does not exist in the whole dataset. I found the following:
0.011
0.014
0.022
0.029
as you noticed the values in the dataframe are rounded. However, in the sampling it got different numbers. Like for example instead of having 0.011, I got the value 0.011000000000000001 in the sample. So My question is how is this even possible? and how to avoid it while sampling? given that I need the values of my output sample to be exactly as the whole data values
EDIT: the type of the output is float64