Is there a way of publishing to SNS when an AWS Glue workflow completes?

user_4214684 0
Viewed 1338

I'm currently using an event rule to publish a message to an SNS topic when a glue job succeeds, like so:

JobCompletedEventRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Sends glue job completed events to SNS topic
      EventPattern:
        source:
          - aws.glue
        detail-type:
          - Glue Job State Change
        detail:
          jobName:
            - !Ref Job
          state:
            - SUCCEEDED
      State: ENABLED
      Targets:
        - Arn: !Ref SnsTopic
          Id: !GetAtt SnsTopic.TopicName

This works well. However, this job is about to become part of a glue workflow, and I now want the SNS message to be published when the workflow succeeds, instead of the individual job. But I can't find a way to do this.

Automating AWS Glue with CloudWatch Events shows the CloudWatch events that are generated by AWS Glue, but there aren't any for workflows.

aws-glue amazon-web-services
2 Answers

I know you asked this question 1 year ago, but I found myself having the same need and I resolved it by adding a "dummy job" (that does nothing) at the end of the workflow and then add a rule similar to yours on SUCCESS of the dummy job.

user_29542560

I have used the boto3 library with the publish() method in the last job of my Glue workflow. That way you can customize the message sent. Useful if you have multiple parallels workflows using the same glue jobs and need to distinguish between them in sns messages.

user_146814210
Related