How to store data extremely quickly

user_13153023 0
Viewed 95

I have to sample a sensor on a raspi at at rate of 1 million samples/second for around 30 seconds. Each sample is 6 characters How do I store this data without slowing down the program? I can save it in memory until after the sample time, or store it directly to the disk. Any help would be appreciated.

raspberry-pi storage python
1 Answers

Not to slow down your Python program you can write to file asynchronously in a separate threading.Thread and use queue.Queue to exchange data with thread. Also writing large portions of data at a time to file will improve speed greatly.

I made a simple Writer class down below that does all above suggestions. After class there are few lines of code with example of usage. This example writes about 1 million samples within just 1 second.

Try it online!

class Writer:
    import threading, queue
    def __init__(self, file_name):
        self.fname = file_name
        self.buf = []
        self.q = self.queue.Queue(128)
        self.t = self.threading.Thread(
            target = self._FileWrite)
        self.t.start()
    def Finish(self):
        self.q.put(self.buf)
        self.buf = []
        self.q.put(None)
        self.t.join()
    def Write(self, obj):
        self.buf.append(obj)
        if len(self.buf) >= 50_000:
            self.q.put(self.buf)
            self.buf = []
    def _FileWrite(self):
        with open(self.fname, 'wb') as f:
            while True:
                buf = self.q.get()
                if buf is None:
                    break
                data = b''
                for e in buf:
                    data += e.encode('ascii')
                f.write(data)

# Example of usage

writer = Writer('out.bin')
for i in range(1_234_567):
    writer.Write('abcdef')
writer.Finish()

import os
print('Resulting file size:', os.path.getsize('out.bin'))

Output:

Resulting file size: 7407402
user_9415310
Related