glue.driver.aggregate.numFailedTasks CloudWatch metric doesn't log failure when Glue job fails

user_214518 0
Viewed 563

I have a Glue PySpark job which has failed after approx 2 mins with an AnalysisException and a Run Status of Failed. I want to detect it for a CloudWatch alarm. In CloudWatch metrics I'm expecting it to record a count of 1 over the period, but it's recording 0. I'm using the following metric config:

   MetricName: glue.driver.aggregate.numFailedTasks
   JobRunId: ALL
   JobName: <MyGlueJobName>
   Statistic: Sum
   Period: 1 min
   Type: Count

It appears other metrics are working e.g. glue.driver.jvm.heap.used but not numFailedTasks. Am I doing something wrong?

amazon-cloudwatch aws-glue amazon-web-services
2 Answers

We had a similar problem/requirement with our glue jobs. We have many jobs and need to monitor success and failure. We've also experienced that the built-in metrics aren't really reliable. But even if they were, they don't really answer the question of whether or not a job was successful. We found a good way for us by generating custom metrics in a generic way for all jobs. This also works for existing jobs afterwards without having to change the code. I wrote an article about it: https://medium.com/@ettefette/metrics-for-aws-glue-jobs-as-you-know-them-from-lambda-functions-e5e1873c615c

user_193471780

The only way that I managed to get around this problem was to create a custom metric. I created a metric using put_metric_data. I call it when an exception is raised in the pyspark script.

user_2145180
Related