Thread safe copying of a deque without blocking append

user_7757891 0
Viewed 338

I'm trying to create a circular buffer in python. My best implementation is by using deque with fixed length.
(basic implementation)

class RingBuffer(object):
    def __init__(self, size):
        self.mutex = Lock()
        self.deque = collections.deque(maxlen=size)

    def push_elem(self, element):
        copy.deepcopy(element)
        with self.mutex:
            self.deque.append(element)

    def get_data(self, event, callback=None):
        with self.mutex:
            return copy.deepcopy(list(self.deque))


    def get_elem(self):
        if self.deque:
            with self.mutex:
                return cp.deepcopy(self.deque[-1])
        return None

I know that that appending and deleting elements from both sides is thread safe but I also need to be able to copy N-number of elements. This is why I'm using this mutex.
The problem is that I don't want to block the insertion of new elements while some other thread is copying the data in the buffer.
Is there a way to achieve this behavior without the mutex?

deque multithreading python
1 Answers

In CPython, this copy is atomic (once it starts to run):

s = list(islice(some_deque, 20))

The list building, slicing with itertools, and deque iteration are all implemented in C with no pure python callbacks. Also, the copy operation doesn't cause any DECREFs to fall to zero, thus eliminating another source of non-atomic behavior.

user_4244990
Related