I have this small python loop running in a thread. It's job is to take new samples coming in from a deque and add them on to the end of a numpy array. It works fine for a while but after about 40,000 samples it starts to slow down considerably. I put in some timing functions at around 41,300 it takes 50ms to run, but not too long after that it takes about 6seconds etc. I imagine this has something to do with me copying the vstack results? Is there a cleaner way to do this?
This Deque is defined in another file and passed into the thread (producer consumer model)
raw_pred_queue = Deque[np.array]()
The loop in question:
raw_pred = np.empty((0,3), float)
og_pred = np.empty((0,3), float)
loop = 0
start = time.time()
while raw_pred_queue and loop < 400:
loop +=1
val = raw_pred_queue.popleft()
raw_pred = np.vstack((raw_pred, val))
og_pred = np.vstack((og_pred, val))