I'm using S3 event based triggers to trigger lambda functions. It triggers a lambda function every time an _SUCCESS file is written at a specific location in S3. Data is being written at the source location using Databricks spark jobs. It has been observed that once the job writes data to source location, the lambda function is trigger twice, consistently.
This behavior is only observed when _SUCCESS is written by Databricks job. I tried to write the file from CLI, it triggers the lambda function just once.
It would be helpful to know the reason behind such a behavior from Databricks jobs.