How can I lock a complex operation that uses database read, insert and web access to protect from concurrent access?

Viewed 159

I am building a cache mechanism and it goes like this:

  • Try to get a certain field from the database
  • If it exists then return it
  • If not, then request it from an external API (It should take 500 ms)
  • Write it to the database & return it

Example:

    async def _get_poly_id(self, longitude: str, latitude: str):
        poly_id = self._polygon_manager.get_poly_id(longitude=longitude, latitude=latitude)
        if poly_id == None:
            geo_json = self._point2geojson(longitude=longitude, latitude=latitude)
            poly_id = await self._agro_api_helper.create_polygon(geo_json) # It can raise exception
            self._polygon_manager.save_poly_id(longitude=longitude, latitude=latitude, poly_id=poly_id)
        return poly_id

It works well until you try to call this (cache) function concurrently in a non already cached field. At that time:

  • All the concurrent requests will check the database
  • All they found that there is no cache
  • All will call the external API
  • The first one will write to the database and return successfully
  • The others will try to write again in the database but will get an integrity error (since there is uniqueness in the field)

One workaround would be to handle the Integrity Error and re-check the db. The problem is that the call to the external API costs and must be avoided.

So. Is there a way to lock this operation and make other process wait? I am using SQLite / SQLAlchemy / FastAPI / Python.

0 Answers
Related