Recognition of a plateau with a slope close to zero

Viewed 34

I am writing code to remove plateau outliers from time series data. I proceeded after receiving advice to use np.diff, but there was a problem that it could not be recognized if it was not the same value.

def find_plateaus(F, min_length=200, tolerance = 0.75, smoothing=15):

    import numpy as np
    from scipy.ndimage.filters import uniform_filter1d
    
    # calculate smooth gradients
    smoothF = uniform_filter1d(F, size = smoothing)
    dF = uniform_filter1d(np.gradient(smoothF),size = smoothing)
    d2F = uniform_filter1d(np.gradient(dF),size = smoothing)
    
    def zero_runs(x):

        iszero = np.concatenate(([0], np.equal(x, 0).view(np.int8), [0]))
        absdiff = np.abs(np.diff(iszero))
        ranges = np.where(absdiff == 1)[0].reshape(-1, 2)
        return ranges
    
    # Find ranges where second derivative is zero
    # Values under eps are assumed to be zero.
    eps = np.quantile(abs(d2F),tolerance) 
    smalld2F = (abs(d2F) <= eps)
    
    # Find repititions in the mask "smalld2F" (i.e. ranges where d2F is constantly zero)
    p = zero_runs(np.diff(smalld2F))
    
    # np.diff(p) gives the length of each range found.
    # only accept plateaus of min_length
    plateaus = p[(np.diff(p) > min_length).flatten()]


    return (plateaus)


plateaus = find_plateaus(test, min_length=5, tolerance = 0.02, smoothing=11)

plateaus = np.ravel(plateaus, order = 'A')

plateaus = plateaus.tolist()
print(plateaus)
test2['T&F'] = np.nan

for i in test2.index:
        if i in plateaus:
            test2.loc[i,['T&F']] = test2.loc[i,'data']
        else :
            test2.loc[i,['T&F']] = 0

fig, ax = plt.subplots(figsize=(15,6))
ax.plot(test2.index, test2['data'], color='black', label = 'time_series')
ax.scatter(test2.index,test2['T&F'], color='red', label = 'D910')
plt.legend()
plt.show();

Do you know any libraries or methods that can be used? I want to recognize the parts marked in the picture below.

enter image description here

1 Answers

Still in progress, but found the answer.

First, make the np array multidimensional.

ex) time_step = 3 .....

Then, using np.std(), find the standard deviation,

After checking, you can set the standard deviation range to recognize the included range.

Related