The way to write ten thousand data points to InfluxDB per second

user_14432029 0
Viewed 837

I’m using a raspberry pi 4 to collect sensor data with a python script.

Like:

val=mcp.read_adc(0)

Which can read ten thousand data per second.

And now I want to save these data to influx for real-time analysis.

I have tried to save them to a log file while reading, and then use telegraf to collect as this blog did:

But it’s not working for my stream data as it is too slow.

Also I have tried to use python's influxdb module to write directly, like:

client.write(['interface,path=address,elementss=link value=3.14'],{'db':'db'},204,'line')

It's worse.

So how can I write these data into influxdb in time. Are there any solutions?

Thank you much appreciated!

Btw, I'm a beginner and can only use simple python, so sad.

influxdb python
1 Answers

InfluxDB OSS will process writes faster if you batch them. The python client has a batch parameter batch_size that you can use to do this. If you are reading ~10k points/s I would try a batch size of about 10k too. The batches should be compressed to speed transfer.

The write method also allows sending the tags path=address,elementss=link as a dictionary. Doing this should decrease parsing effort.

Are you also running InfluxDB on the raspberry pi or do you send the data off the Pi over a network connection?

I noticed that you said in the comments that nanosecond precision is very important but you did not include a timestamp in your line protocol point example. You should provide a timestamp yourself if the timing is this critical. Without an explicit timestamp in the data, InfluxDB will insert a timestamp at "when the data arrives" which is unpredictable.

As noted in the comments, you may want to consider preprocessing this data some before sending it to InfluxDB. We can't make a suggestion without knowing how you are processing the piezo data to detect footsteps. Usually ADC values are averaged in small batches (10 - 100 reads, depending) to reduce noise. Assuming your footstep detector runs continuously, you'll have over 750 million points per day from a single sensor. This is a lot of data to store and postprocess.

Please edit your question to include move information, if you are willing.

user_35160340
Related