Arranging real-time sensor input data based on timestamp

user_12296586 0
Viewed 113

I have real-time time series data from two different sensor input. I would like to arrange the data according to the timestamp.

The problem is that the sensor input is recieved inconsistent, e.g.

 sensor      time  data1     data2     data3     data 4  
['sensor_1', '0', '0.1234', '0.3251', '1.4235', '0.4234']
['sensor_1', '1', '0.1432', '0.3452', '1.4245', '0.1434']
['sensor_2', '0', '0.6543', '0.3231', '1.1235', '0.3434']
['sensor_1', '2', '0.1654', '0.3243', '1.5445', '0.1234']
['sensor_2', '1', '0.1544', '0.3345', '1.5425', '0.6534']
['sensor_2', '2', '0.5432', '0.4551', '1.8755', '0.4245']

How can I join them (while I recieve data) so sensor data with same timestamp are combined, like so:

 time  data1_1   data1_2   data1_3   data1_4   data2_1   data2_2   data2_3   data2_4    
['0', '0.1234', '0.3251', '1.4235', '0.4234', '0.6543', '0.3231', '1.1235', '0.3434']
['1', '0.1432', '0.3452', '1.4245', '0.1434', '0.1544', '0.3345', '1.5425', '0.6534']
['2', '0.1654', '0.3243', '1.5445', '0.1234', '0.5432', '0.4551', '1.8755', '0.4245']

I recieve data through a subprocess (one line at a time):

process = subprocess.Popen(['node', 'server.js'], stdout=subprocess.PIPE)
while True:
     Byte_data = process.stdout.readline()
     string_data = byte_data.rstrip().decode("utf-8") # From byte to string
     list_data = string_data.split(",") # Comma separate strings

     # Sort data here...
real-time sorting python
2 Answers

You will need to keep a look back and match timestamps as they come in, something like this.

  • From the formulation of your question, it seems the timestamps are matching. If they are not, but only ordered, things get a bit more complicated.
  • If some frames are missing, they will linger indefinitely in the lookback dicts. If this thing is running for long periods of time, you might want to clear out old frames regularly.
def time(row):
    return row[1]

def synchronized(rows):
    lookback = {'sensor_1': {}, 'sensor_2': {}}
    for row in rows:
        # add row to lookback
        sensor = row[0]
        lookback[sensor][time(row)] = row

        # check for matching timestamps
        rms = []
        for t, r in sorted(lookback['sensor_1'].items(), key=time):
            if t in lookback['sensor_2']:
                yield r, lookback['sensor_2']
                rms.append(t)

        for rm in rms:
            for tmp in lookback.values():
                del tmp[rm]

def parse(rows):
    for row in rows:
        yield tuple(row.decode().rstrip().split(','))

import subprocess
process = subprocess.Popen(['node', 'server.js'], stdout=subprocess.PIPE)
for row1, row2 in synchronized(parse(process.stdout)):
    print(row1, row2)
user_3634370

Seeing as your timestamps are all integers you can use them as indexes for a sorted list and append or add data as it comes in at the index equal to the timestamp.

sorted_data = []

process = subprocess.Popen(['node', 'server.js'], stdout=subprocess.PIPE)
while True:
    Byte_data = process.stdout.readline()
    string_data = byte_data.rstrip().decode("utf-8") # From byte to string
    list_data = string_data.split(",") # Comma separate strings

    try:
        sorted_data[int(list_data[1])] += list_data[2:]
    except IndexError:
        sorted_data.append(list_data[1:])

    # prints sorted data in real time
    print(str(sorted_data).replace("],", "],\n"))

# prints out final list with sorted data
print(str(sorted_data).replace("],", "],\n"))
user_144768260
Related