Just wondering if anyone has any thoughts on an approach for orchestrating the transformation steps within a Spark job by using Airflow.
Most Airflow demos that include Spark jobs seem to follow the same kind of process of defining a DAG that looks something like this:
> A: trigger to pick up a file from an S3
> B: move the file to a different S3 location
> C: run a spark job to create an aggregate extract
> D: send the extract out, e.g. via email
What I'm trying to understand are the possibilities around also using Airflow to orchestrate the inner workings of the Spark job itself. For simplicity, an ETL spark job might look something like
> C1: spark.read.csv(xxxx)
> C2: transform dataframe with function transform_1()
> C3: transform dataframe with function transform_2()
> C4: transform dataframe with function transform_3()
> C5: write dataframe to S3
All those C steps need to run inside the same spark job because they are operating on the same Spark DAG. If each of the tasks in the job were considered as part of the Airflow DAG, the flow would be A, B, C1, C2, C3, C4, C5, D.
Depending on the data being processed, a different selection of transform functions might be required. It would be great if we could define the transform functions as Airflow tasks, so for a different set of data, the DAG might be A, B, C1, C2, C4, C5, C7, D.
Is there a way to define a DAG and effectively delegate a sub-DAG to spark and have the spark driver be aware of and be able to communicate back to Airflow to identify which function to run next and feed back updates to Airflow so the DAG is updated with progress?
NB: I know that, in many cases, the calling of transform functions in pyspark will be fast as it's just building a spark DAG and won't actually take any time until the spark DAG is actioned - the purpose of this is question is to define the composition of the spark DAG by using Airflow and be able to track its progress. I also know that, if one of the C tasks fails in spark, in most cases, all the preceding C tasks (i.e. the whole sub-DAG) would need to be re-run.
Many thanks for reading and I look forward to hearing your thoughts.
Dave