Adding and removing values from list python

sumit mandal 0
Viewed 46

I have continous stream of data coming in from raspberry pi and I am appending them in a list. I am predicting the output in realtime, the thing now is Memory Management, I am afraid if keep the device running continously it will fill the list and eventually I may run out of memory. Is there any way I can append the value predict them and keep removing all the values from the list that are 100th prediction (index) above my current prediction(index).

i2c = board.I2C()
mpu = adafruit_mpu6050.MPU6050(i2c)
mpu_accelerometer_range = adafruit_mpu6050.Range.RANGE_4_G

data1 = []
j = 0
data2 = []
while(True):
    
    
    j+=1
    ax,ay,az = mpu.acceleration
    magnitude = np.sqrt(np.square(ax)+np.square(ay)+np.square(az))
    data1.append([magnitude])
    df = pd.DataFrame(data1)
memory-management append list python
2 Answers

I think a solution is to predict the values for the first 100 incoming pieces of data and then start the 'while True'-loop in which you delete the last item of the list at the end of the loop. I'm not familiar with the library and functions you use, but I think it might look like this:

i2c = board.I2C()
mpu = adafruit_mpu6050.MPU6050(i2c)
mpu_accelerometer_range = adafruit_mpu6050.Range.RANGE_4_G

data1 = []
j = 0
data2 = []

for i in range(100):
    j+=1
    ax,ay,az = mpu.acceleration
    magnitude = np.sqrt(np.square(ax)+np.square(ay)+np.square(az))
    data1.append([magnitude])
    df = pd.DataFrame(data1)

while(True):
    j+=1
    ax,ay,az = mpu.acceleration
    magnitude = np.sqrt(np.square(ax)+np.square(ay)+np.square(az))
    data1.append([magnitude])
    df = pd.DataFrame(data1)
    
    data1.pop(0)

Using the pop function and specify the first (in python index 0) element of the list we can remove the element. This might seem confusing, but look at it this way: the oldest item of the list is always the first one. And because we did the loop 100 times at the beginning, you are now constantly deleting the 100st item of the list.

Let me know if this wasn't a solution or if you have any questions.

JvO-dash0

You currently do this:

data1 = []
j = 0
while(True):
    j+=1
    magnitude = ... #something
    data1.append([magnitude])
    df = pd.DataFrame(data1)

Instead, if there's a maximum number n of values you need to store, you could do this:

data1 = [None] * n
j = 0
while True:
    j+=1
    magnitude = ... #something
    data1[(j - 1) % n] = [magnitude]
    df = pd.DataFrame(data1[j % n:] + data1[:j % n if j % n else n])

This will keep the memory footprint at a fixed size on the order of n, and at each iteration will deliver a dataframe with the n most recent values of your measurement in chronological order.

Or you could optimize even further than in the above approach by eliminating data1 and updating the dataframe logic to simply do this:

# before entering loop:
df = pd.DataFrame([[None] for _ in range(n)])

# inside loop:
df.iloc[(j - 1) % n, 0] = magnitude

In both of the above approaches (with and without data1), you would either want the downstream logic to be aware that trailing null values in df indicate unset data (for values of j that are < n) or include the number of meaningful values, namely min(j, n), somewhere in your output.

constantstranger0
Related