Python: Get time of beginning and end of curve in time series linear graph

user_7062510 0
Viewed 195

I am trying to find the beginning and the end of a linear curve. There might be some dips in the curve, so there should be a "if less than x for y seconds, then ignore"

the flat line at beginning and end is not 0, but is a measurement of the background noise from the sensor.

I am sure that there is a name for this in scipy or something, but thus far, I have been unable to find it, so can anyone help (not necessarily in scipy or pandas)

code and graph example below. I need to know the time (in nanoseconds) for T1 and T2, so I can find the total time for the measurement.

import pandas as pd
dg = pd.read_csv("curve.csv", sep=";")

dg.to_json()
'{"tol":{"0":"0,60","1":"0,40","2":"0,60","3":"0,50","4":"0,60","5":"0,40","6":"0,60","7":"0,40","8":"0,50","9":"0,70","10":"0,40","11":"14,00","12":"15,00","13":"13,00","14":"16,00","15":"16,00","16":"12,00","17":"1,00","18":"1,00","19":"19,00","20":"18,00","21":"17,00","22":"16,00","23":"19,00","24":"30,00","25":"20,00","26":"5,00","27":"0,50","28":"0,60","29":"0,40","30":"0,50","31":"0,30","32":"0,70","33":"0,30"},"time":{"0":"06:00:01","1":"06:00:02","2":"06:00:03","3":"06:00:04","4":"06:00:05","5":"06:00:06","6":"06:00:07","7":"06:00:08","8":"06:00:09","9":"06:00:10","10":"06:00:11","11":"06:00:12","12":"06:00:13","13":"06:00:14","14":"06:00:15","15":"06:00:16","16":"06:00:17","17":"06:00:18","18":"06:00:19","19":"06:00:20","20":"06:00:21","21":"06:00:22","22":"06:00:23","23":"06:00:24","24":"06:00:25","25":"06:00:26","26":"06:00:27","27":"06:00:28","28":"06:00:29","29":"06:00:30","30":"06:00:31","31":"06:00:32","32":"06:00:33","33":"06:00:34"}}'

import plotly.graph_objects as go
fig = go.Figure(data=go.Line(y=dg.tol.to_list(),x = dg.time))
fig.show()

enter image description here

pandas plotly time-series python
1 Answers

So I would implement a moving average and then get the min and max timestamps when the rolling average is above a certain value.

The following code should help:

import numpy as np
import pandas as pd
np.random.seed(1)

# Create fake data that has low levels besides a model with high values
rows,cols = 1000,1
data = np.random.rand(rows,cols) # You can use other random functions to generate values with constraints
tidx = pd.date_range('2019-01-01', periods=rows, freq='S') # freq='MS'set the frequency of date in months and start from day 1. You can use 'T' for minutes and so on
data_frame = pd.DataFrame(data, columns=['value'], index=tidx)
data_frame['row_number'] = range(len(data_frame))
data_frame['value'] = np.where(
    (data_frame['row_number'] >= 250) & (data_frame['row_number'] <= 500),
    (data_frame['value'] + .02) * 1000,
    data_frame['value'] * 50
)
data_frame = data_frame.drop('row_number', axis=1)

# Tune the below two values to get the proper senstivity
rolling_string = '5s'
trigger_amount = 75

# Run rolling average
data_frame['rolling_average'] = data_frame.rolling(rolling_string).mean()
data_frame = data_frame.reset_index()
data_frame.columns=['date', 'value', 'rolling_average']

# Get rows of the trigger amount
data_frame['increased'] = np.where(
    data_frame['value'] >= trigger_amount,
    True,
    False,
)

# Find the earliest and latest date above these values
early_date = data_frame[data_frame['increased'] == True]['date'].min()
late_date = data_frame[data_frame['increased'] == True]['date'].max()

# Print values
print("t1: " + str(early_date) + "\nt2: " +str(late_date))

###Output###
t1: 2019-01-01 00:04:11
t2: 2019-01-01 00:08:20

The more seconds (or in your case nanoseconds) you use for the rolling average the slower it will be to pick up on sharp increases and decreases. The same goes for the trigger value as well.

And finally to visualize my work:

import plotly.express as px

viz = data_frame.melt(id_vars=['date'], value_vars=['value', 'rolling_average'])

fig = px.line(data_frame=viz, x='date', y='value', color='variable')
fig.add_vline(x=early_date)
fig.add_vline(x=late_date)
fig.add_hline(y=trigger_amount)
fig.show()

enter image description here

user_48291580
Related