I would like to verify if there are outliers in the column of a dataframe knowing that it must be increasing or decreasing
For example the dataframe df_bad is growing and its data is the following:
df_bad = pd.DataFrame([1,2,-3,1,5,8,7],index=[0,10,20,30,40,50,60])
The correct dataframe should be corrected to:
df_correct = pd.DataFrame([1,2,3,4,5,6,7],index=[0,10,20,30,40,50,60])
where it is identified that the indices 20,30 and 50 are incorrect because the value of the row df_bad.loc[20] is less than the value of the previous row, the value of the row df_bad.loc[30] is greater than its previous row, however, is not greater than the highest correct value (df_bad.loc[10]), the value of row df_bad.loc[50] is greater than its previous row, however, it is also greater than its next row.
If possible I would like to correct the df_bad dataframe as best as possible to the df_correct dataframe. It should be noted that only in the example the increasing steps are one at a time, and that the dataframe that I want to correct has more columns with the same trend.
If it cannot be correctly corrected, a df_checked dataframe could be created where the atypical data is changed to zero and is subsequently filled in with the average of the adjacent ones or, if possible, with the trend to create the dataframe df_fixed.
df_checked = pd.DataFrame([1,2,0,0,5,0,7],index=[0,10,20,30,40,50,60])
df_fixed = pd.DataFrame([1,2,3.5,3.5,5,6,7],index=[0,10,20,30,40,50,60])