How to process an array of objects containing sensor readings and timestamps and validate how many times and how long the sensor had a certain value?

Viewed 55

I am processing some sensor data and my input looks something like this (the values are always 1 or 0, but the length of the values & timestamps arrays is a lot longer, as they contain the readings of a sensor for 24 hours & the data ingestion happens every second):

 const input=   {
  values: [ 0, 1, 0, 0, 1 ],
  timestamps: [
    '2022-08-19T08:01:21.000Z',
    '2022-08-19T08:01:22.000Z',
    '2022-08-19T08:01:23.000Z',
    '2022-08-19T08:01:24.000Z',
    '2022-08-19T08:01:25.000Z'
  ]
}

or I could easily convert the input to the following format (I couldn't decide, which one would be more suitable):

 const input=  [{value: 0, timestamp: '2022-08-19T08:01:21.000Z'},
                {value: 1, timestamp: '2022-08-19T08:01:22.000Z'},
                {value: 0, timestamp: '2022-08-19T08:01:23.000Z'},
                {value: 0, timestamp: '2022-08-19T08:01:24.000Z'},
                {value: 1, timestamp: '2022-08-19T08:01:25.000Z'}]

My goal is to identify all the periods when the sensor reading was 0 and also validate if these periods were shorter or longer than 1 minute, i.e., for the case above, I'd like to get the following result:

result = [
    {startDate: '2022-08-19T08:01:21.000Z', endDate= '2022-08-19T08:01:22.000Z', isShorterThanOneMin: true},
    {startDate: '2022-08-19T08:01:23.000Z', endDate= '2022-08-19T08:01:25.000Z', isShorterThanOneMin: true}]

Could you advise a time-efficient way to solve this?

3 Answers

Simple JavaScript version

class Processor {
    constructor(inputs) {
        this.inputs = inputs;
        this.outputs = [];
        this.process();
    }
    process() {
        while (this.inputs.length > 0) {
            const input = this.inputs.shift();
            if (this.previousInput === undefined) {
                this.previousInput = input.value === 0 ? input : undefined;
                continue;
            }
            if (this.previousInput.value === 0) {
                if (input.value === 0)
                    continue;
                this.outputs.push({
                    startDate: this.previousInput.date,
                    endDate: input.date,
                    isShorterThanOneMinute: (input.date.getTime() - this.previousInput.date.getTime()) < 60000
                });
                this.previousInput = undefined;
            }
        }
    }
}
const inputs = [
    { value: 0, date: new Date("2022-08-19T08:01:21.000Z") },
    { value: 1, date: new Date("2022-08-19T08:01:22.000Z") },
    { value: 0, date: new Date("2022-08-19T08:01:23.000Z") },
    { value: 0, date: new Date("2022-08-19T08:01:24.000Z") },
    { value: 1, date: new Date("2022-08-19T08:01:25.000Z") }
];
const processor = new Processor(inputs);
console.log(processor.outputs);

Fancier, longer TypeScript version

interface Input {
    value: number;
    date: Date;
}
namespace Input {
    export type List = Input[];
    export const clone = (input: Input): Input => {
        return {
            value: input.value,
            date: new Date(input.date.getTime())
        }
    }
}
interface Output {
    startDate: Date;
    endDate: Date;
    isShorterThanOneMinute: boolean;
}
namespace Output {
    export type List = Output[];
    export const clone = (output: Output): Output => {
        return {
            startDate: new Date(output.startDate.getTime()),
            endDate: new Date(output.endDate.getTime()),
            isShorterThanOneMinute: output.isShorterThanOneMinute
        }
    }
}
class Processor {
    private previousInput?: Input;
    private outputs: Output.List = [];
    private inputs: Input.List;
    constructor(inputs: Input.List) {
        this.inputs = inputs.map(Input.clone);
        this.process();
    }
    private process() {
        while (this.inputs.length > 0) {
            const input = this.inputs.shift()!;

            if (this.previousInput === undefined) {
                this.previousInput = input.value === 0 ? input : undefined;
                continue;
            }

            if (this.previousInput.value === 1) {
                throw new Error(`This is not possible, because we never store an input with value = 1.`);
            }

            if (input.value === 0) continue;

            this.outputs.push({
                startDate: this.previousInput.date,
                endDate: input.date,
                isShorterThanOneMinute: (input.date.getTime() - this.previousInput.date.getTime()) < 60000
            });

            this.previousInput = undefined;
        }
    }
    getOutputs(): Output.List {
        return this.outputs.map(Output.clone);
    }
    append(input: Input): this {
        this.inputs.push(Input.clone(input));
        this.process();
        return this;
    }
}

const inputs: Input.List = [
    { value: 0, date: new Date("2022-08-19T08:01:21.000Z") },
    { value: 1, date: new Date("2022-08-19T08:01:22.000Z") },
    { value: 0, date: new Date("2022-08-19T08:01:23.000Z") },
    { value: 0, date: new Date("2022-08-19T08:01:24.000Z") },
    { value: 1, date: new Date("2022-08-19T08:01:25.000Z") }
];

const processor = new Processor(inputs);
console.log(processor.getOutputs());

// Continue using the instance as more entries because available...
processor.append({ value: 1, date: new Date("2022-08-19T08:02:25.000Z") });
processor.append({ value: 1, date: new Date("2022-08-19T08:03:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:04:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:05:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:06:25.000Z") });
processor.append({ value: 1, date: new Date("2022-08-19T08:07:25.000Z") });
console.log(processor.getOutputs());

EDIT - @focorner has the fastest code for it methodsName other. Source: https://sourceb.in/dXDPkOliQr [Made few changes.]

@focorner's answer should be accepted answer as the difference is too high. Idk If I am doing something wrong, but the difference in unbelievable

enter image description here

Ignore -

The current fastest way would be the one I tested called methodTwo. I tested answers from here, and the answer from @focorner doesn't work, not sure why, Maybe due to timestamps.

Of the two methods that I wrote, first, one called Process is actually O(n) time complexity and it's slower than the second version of its. The method processTwo is the fastest method. Note that, these tests were performed on intel i9 with 128GB Ram with an input array length of 360_000.

Having the timestamps as number is the fastest way to do this as new Date().getTime() or Date.now() are going to make it slower. Date.now() is faster than Date().getTime()

Also destructuring the objects like { value, timestamp } = input is going to increase the time. Destructured assignments are less computationally efficient than traditional.

I used a package called benchmark to test the methods.

processTwo works on very simple logic, that is it checks two elements in a single loop ie element on index i and i + 1.

Json Source: https://rentry.co/x9vtb [360000 is too huge for internet, so smaller bersion] Source: https://sourceb.in/c9garaf4Ld enter image description here

The seconds test took only the test data provided by the author and here processTwo is the fastest. I might be wrongly testing but I tried and found out processTwo faster.

I also saw that the output of the Author's approach is just { start: Date, end: Date } and doesn't actually calculate isShorterThanOneMin so I added it to make it even.

You should start storing the number timestamps in an array instead of ISO date or strings. Source: https://sourceb.in/vqTq2dBMmT enter image description here

I figured it out, not sure if it's the fastest way, but seems to work:

const input=  [{value: 1, timestamp: '2022-08-19T08:01:21.000Z'},
                {value: 1, timestamp: '2022-08-19T08:01:22.000Z'},
                {value: 0, timestamp: '2022-08-19T08:01:23.000Z'},
                {value: 0, timestamp: '2022-08-19T08:01:24.000Z'},
                {value: 0, timestamp: '2022-08-19T08:01:25.000Z'},
                {value: 1, timestamp: '2022-08-19T08:01:29.000Z'}]


let result = input.reduce((finalArray, currentValue,index, arr) => {
  if (currentValue.value === 0) {
    let resultElement = {};
    resultElement.start = currentValue.timestamp;
    
    //find end Date
    let nextToCheck = index;
    let endFound = false;
    do {
     nextToCheck = nextToCheck + 1;
    if (nextToCheck === arr.length) {
       break;
     }
     if ((arr[nextToCheck].value) === 1) {
       resultElement.end = arr[nextToCheck].timestamp;
       endFound = true;
     }
     
    } while (!endFound)
  
   // find out if previous one was 1 --> if it was 0, it should not be pushed to avoid having sub-arrays
     if (index !== 0 && arr[index - 1 ].value !== 0) {
        finalArray.push(resultElement)
     }  
     if (index === 0) {
        finalArray.push(resultElement)
     }
  
  }
  return finalArray;
   
  
}, [])
console.log(result)
Related