Problem: We have an AWS Lambda function which calls 3rd party API. That 3rd party system has some concurrency issue so under some circumstances if we call that API from several instances of the lambda function, which are running at the same time, we end up duplicated objects created in 3rd party system. Vendor of that system is not promising to fix that issue quickly so for the moment we need to avoid concurrency of the section of the code which calls that API
Our Lambda function is triggered when file is dropped into S3 bucket, so when multiple files are dropped at the same time, each of these files is processed by a running an instance of this function in parallel.
For the performance reasons we would like to continue processing files concurrently, so we do not consider a solution using a queue (like SQS) to put the files in queue and then call lambda function for each of them sequentially.
What is the easiest way to write a kind of a
threading.lock()
command to block that critical section of the code that calls 3rd party API from executing in parallel instances?
We think that we can probably implement that by updating a value in DynamoDB and checking if it is locked or not, but at the moment we don't use DynamoDb , our solution contains S3 buckets and serverless components like AWS Step Functions, Lambda functions, Glue jobs, so introducing a database just for a single purpose of controlling locks sounds like an overkill
Any thoughts are welcome