Fixed Size Persisted to Disk Queue using queuelib

user_6765276 0
Viewed 37

I'm trying to use queuelib github to implement a stored in disk (persisted to disk) fixed size queue.

Code snapshot:

from queuelib import FifoDiskQueue
images_data = FifoDiskQueue(r"c:\temp")

while True:
    # doing something
    frame = np.random.rand(3,2)
    # implement fixed size queue
    if len(images_data) >= 10:
        images_data.pop()
    images_data.push(frame.tobytes())  # add the new frame to the main database

# after the while loop ends
images_data.close()  # close the handle to the queue

While running it for long time I observed that:

  1. Queue file q00000 is always 0 KB
  2. Device's disk memory is decreasing until it reach the limit of the PC memory. I think that the queue is keep getting bigger and bigger although I'm "pop-ing" out any old element before adding new ones...

I'm using Windows with Python 3.8

Can some help me figure out why the size keep getting bigger or point me to the right way to implement a fixed size queue?

Thank you!!!

scrapy persistence queue python-3.x
1 Answers

So after looking through the source code of queuelib it appears that what queuelib actually does when you add and remove records from the persistent disk storage, what it is actually doing is keeping track of an internal offset and adding or subtracting from it while the Queue remains open.

This means that every record that you add to the queue is written to the file and remains written to the file until the queue is closed at which point it discards any data that you previously popped from the data structure.

So one possible solution would be to close and then reopen the queue once in a while.

For example,

from queuelib import FifoDiskQueue
import numpy as np

path = "C:\\temp"
data = FifoDiskQueue(path)

counter = 0
while True:
    frame = np.random.rand(2,3).tobytes()
    data.push(frame)
    if len(data) >= 10:
        data.pop()
    counter += 1
    if counter % 1000 == 0:
        data.close()
        data = FifoDiskQueue(path)

Update

There is also the chunksize argument that could be lowered which essentially would do the same thing as the previous solution when set to a low number. The default is 100,000. Setting this to a lower value is the better solution. The chunksize is a reference to the number of records kept in the storage before it closes the file internally, so a lower number limits the maximum size a file can get before it is closed and popped data is discarded.

For example:

from queuelib import FifoDiskQueue
import numpy as np

path = "C:\\temp"
data = FifoDiskQueue(path, chunksize=1000)
while True:
    frame = np.random.rand(2,3).tobytes()
    data.push(frame)
    if len(data) >= 10:
        data.pop()
user_178294510
Related