I've 6 spark jobs which must execute in an order and are dependent on the previous one and a 7th job which must execute irrespective of the previous jobs. I've set trigger rule as 'ALL_DONE' for my 7th job.This is the exact scenario.
Now what the requirement is say i.e my 3rd job gets failed, my 7th job must know that the job failed is 3. Now i've defined a function at deafult_args as 'on_failure_callback': 'find_failed_job()' which checks the status of all the jobs from begining and returns the exact name of the job failed.
Here come the my issue, i'm able to XcomPush the value with key as 'failed' and value as 'jobName' Now how can i pull this value in my 7th job?
In general, if it was a task, the i can pull using xcom.pull(task_Id,key)....but here it is just a function definition but not a task so what can i give a variable to 'task_Id' in xcom_pull.
As per my client requirement, i shouldn't create a task and call this.If that is the case, i'd have got a solution.
To be precise, how can we pull a variable which was pushed in a function definition.
Quick help is much appreciated. Thanks.