Resource temporarily unavailable on Python shelve.open

user_940299 0
Viewed 4572

I have a Python application that does the following:

  • Is called by another process/processes once every 2-3 minutes in order to store an object using with shelve.open(shelvefilename, flag='c').
  • Is called by another process/processes many times per minute in order to read that shelve file using with shelve.open(shelvefilename, flag='r')

The problem is that at some time I get a _gdbm.error: [Errno 11] Resource temporarily unavailable Error:

   File "/path/to/myprog.py", line 755, in mymethod
    with shelve.open(shelvefilename, flag='r') as shlvfile:
  File "/usr/local/lib/python3.6/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/local/lib/python3.6/shelve.py", line 227, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/local/lib/python3.6/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable

My guess is that this happen because at some moment I have opened the shelve file both for read and write operation which is problematic by definition.

Is there any way that I can make the update to the shelve file without disturbing the read operations?

shelve concurrency python
2 Answers

I just ran into this issue using shelve in a small module for caching exchange rates. You can solve the issue with python threading and a global threading.Lock() object.

from threading import Lock
import shelve

mutex = Lock()

mutex.acquire()
db = shelve.open(db_name)
# write to db
db.close()
mutex.release()

the original source for the code snippet and credit for the idea can be found @ http://georg.io/2014/06/Python-Shelve-Thread-Safety

The ```threading.Lock()`` object works by blocking the other python threads until it's released. see https://docs.python.org/2/library/threading.html#lock-objects

user_25304840

This is more of a conceptional issue. If one process modifies data in a file while another process reads it at the same time, the result is kind of unpredictible.

Imagine that you read a portion of file where only half of some value has been written at this point in time. The read would simply fail to parse the entry correctly and probably all subsequent entries too. In other words, it will break sooner or later.

I think the best approach would be to centralize the "shelve" in a single process or use a database.

user_2747570
Related