How to lock a single S3 object temporarily

user_2092250 0
Viewed 178

I have a single S3 object (file) and multiple Lambdas that have read/write access to it. Most operations are read only, but when writing to the file what I would like to do is something like this:

  1. Lock myfile.json
  2. GetObject("myfile.json")
  3. Edit file contents
  4. PutObject("myfile.json")
  5. Unlock myfile.json

The goal is to ensure that a competing Lambda cannot also call PutObject() while it is being updated.

Is this possible with S3?

amazon-web-services amazon-s3
1 Answers

S3 unfortunately does not offer a native way to handle these situations - it's more of a "blob storage" service, rather than a database. That being said, you have 2 options:

Use a database

You can use e.g. advisory locks, like what Postgres offers. There are a few things to keep in mind when doing that.

  1. Make sure all your processes can access the database
  2. Make sure you keep the session open while you are holding the lock.
  3. There are some gotchas when it comes to working with many processes at the same time. Postgres, for example, has a hard limit for the number of concurrent sessions it allows. If you're working with e.g. a large computational grid, you may want to connect to your PG instance via PGBouncer.

Use an external service

I've seen people use lockable for things like this.

They have a Python client, if you're using Python:

$ pip install lockable-dev

from lockable import Lock

with Lock('my-lock-name'):
   obj = get_object("myfile.json)
   obj = edit_object(obj)
   put_object(obj, "myfile.json")

Alternatively, you can hit their HTTP endpoints directly, for example (using Python, but it's language agnostic):

import requests

has_lock = False

# acquire the lock
while not has_lock:
    res = requests.get('https://lockable.dev/api/acquire/my-lock-name')
    has_lock = res.json(['response'])

# do the work
obj = get_object("myfile.json)
obj = edit_object(obj)
put_object(obj, "myfile.json")

# release the lock
requests.get('https://lockable.dev/api/release/my-lock-name')
user_193453350
Related