I have a large data file that consist of many cycles. I am trying to find some key points for each of these cycles. I would like a way to narrow down the points I have identified but have not been successful.
#Determine Relevant Points A and B
Data3['Diff'] = Data3['Value'].diff(2)
Data3['Diff2'] = Data3.groupby('Cycle#', as_index=False)['Value'].transform(lambda x: x - 2*x.shift(1) + x.shift(2))
Data3['POI'] = Data3.groupby('Cycle#').apply(
lambda g: np.select([g['Value'] == g['Value'].max(),
g['Diff2'] == g['Diff2'].min()],
['A', 'B'], default='-')
).explode().set_axis(Data3.index, axis=0)
I have tried this but would like to refine it more. Right now it goes through my column labeled values, and will get the first derivative and the 2nd derivative. You can see me identifying point A to the max of the value, and B to the min of the 2nd derivative.
I want to refine this more, and in addition to A being the max of the value I want it to also ensure the lag of the 2 values before it is less than the max. So for example, if the max is 1280, ensure the 2 rows behind it are less than this value. This helps get rid of some outliers in the data.
For B i want to get rid of any 2nd derivative values less than -83. This also helps get rid of outliers.
The question is, where would I add this info. Would this be in the equation here:
Data3['POI'] = Data3.groupby('Cycle#').apply(
lambda g: np.select([g['Value'] == g['Value'].max(),
g['Diff2'] == g['Diff2'].min()],
['A', 'B'], default='-')