I am new to python. I have the following code:
data = {'deviceTimestamp' : ['2022-08-21 00:00:04.019345800','2022-08-21 00:00:09.666780800',
'2022-08-21 00:00:15.193316200','2022-08-21 00:00:20.708187',
'2022-08-21 00:00:26.212871600','2022-08-21 00:00:31.792147900',
'2022-08-21 00:00:37.345935900','2022-08-21 00:00:42.855598900',
'2022-08-21 00:00:48.380273','2022-08-21 00:00:53.894567100',
'2022-08-21 00:00:59.421452300','2022-08-21 00:01:04.994086700',
'2022-08-21 00:01:10.549778100','2022-08-21 00:01:16.067442300',
'2022-08-21 00:01:21.585345300','2022-08-21 00:01:27.123348',
'2022-08-21 00:01:32.657100400','2022-08-21 00:01:38.681526500',
'2022-08-21 00:01:44.254180800','2022-08-21 00:01:49.832247900',
'2022-08-21 00:01:55.404809800','2022-08-21 00:02:00.935254300',
'2022-08-21 00:02:06.456698200','2022-08-21 00:02:12.023842500',
'2022-08-21 00:02:17.601805600'],
'Failure_Sensor' : [0,1,1,1,0,0,0,1,0,0,0,1,1,1,0,0,0,0,1,0,1,1,1,0,0]
}
df_failure_sensor = pd.DataFrame(data)
df_failure_sensor['Failure_Sensor_Transition'] = [(df_failure_sensor['Failure_Sensor'][i] if df_failure_sensor['Failure_Sensor'].shift(1)[i] != df_failure_sensor['Failure_Sensor'][i]
else (df_failure_sensor['Failure_Sensor'][i] if((df_failure_sensor['Failure_Sensor'].shift(-1)[i] != df_failure_sensor['Failure_Sensor'][i]))
else ""))
for i in range(0, len(df_failure_sensor))]
df_failure_sensor = df_failure_sensor[df_failure_sensor['Failure_Sensor_Transition'] != ""]
df_failure_sensor.reset_index(inplace=True, drop=True)
df_failure_sensor["Start"] = [(str(df_failure_sensor['deviceTimestamp'][i]) if df_failure_sensor['Failure_Sensor'].shift(-1)[i] == df_failure_sensor['Failure_Sensor'][i] else "")
for i in range(0, len(df_failure_sensor))]
number_of_faults = df_failure_sensor['Failure_Sensor'][(df_failure_sensor['Start'] != "") & (df_failure_sensor['Failure_Sensor'] == True)].count()
To explain to you what I did here: I am trying to get the total number of failures by checking first how did the transition of the column Failure Sensor. Then creating the Start column from devicetimestamp if the next value of Failure Sensor is equal to the value and finally getting the count when the values are true and ignoring the empty timestamps.
The code works perfectly but on a very small dataset.
I am trying to execute this code on a dataset containing 300,000 rows and it is taking too long. Please note that this code is also being used to get the number of faults of several columns. So the execution time will take more in this case
Is there a way to improve this code or to get the same results in a different approach in an efficient way ?