I have AWS Glue job converts parquet to CSV files and in the process retrieve before & after row counts in the gluejob and want to send notification if counts does not match. How to implement this? Planning to do it on Step function.
Step # 1. Execute AWS glue job (at the end of the glue job, If counts does not match then sys.exit('counts does not match') this causes glue job failure.
Step# 2. On Error handling(step#1), Will call lambda function passing job_name, job_runId to check glue job status.
Step# 3. Retrieve glue job run status from lambda function & pass values to SNS notification.
Question - Step#2 & Step#3 even possible? If so, need lambda function code to check glue job run status.
Please help!!!