I'd like to as a general question about running apps on Spark. I have a pyspark app which processes the data for various customers. Processing logic is the same, but data is customer specific. Essentially it is an ETL process on Spark and written in Python. For now I am running in client mode on my dev environment. I am submitting apps via spark-submit. Customer ID is my Spark app name:
$> spark-submit --jars my/jars MySparkAppETL.py CustomerID
My goal is to prevent consequent Spark app execution for a given CustomerID if there is already app running with the same CustomerID?
My current thought is to build required logic myself, something like this: 1. Setup metadata repository 2. Create table tracking apps execution, set app status to 'InProcess' at the start of the app. 3. Create a module which will be checking app status for a specific CustomerID, if a job with this CustomerID is still running, prevent the consequent job from start. 4. Update app status to 'Done' when job completes successfully.
My question is there any mechanism in Apache Spark or in some other readily available tools?
Thank you, Pit.