AWS Glue output to stream

Viewed 19

I'm just starting to get familiar with AWS and it's tools and have been researching Glue/DataBrew. I'm trying to understand if it would fit a streaming use case I have in mind and I can clearly see plenty of documentation around consuming streaming data into Glue, but I can't find anything related to publishing streaming data from a glue job.

What I would like to do is pick up a file from some source, rip it apart into component records using Glue and then publish each individual record onto a stream (Kinesis, SNS, Kafka, etc). Is this yet possible with Glue? or am I barking up the wrong tree here.

Is there a better more appropriate AWS solution for this type of use case?

1 Answers

pick up a file from some source

Use S3... Hook a AWS Lambda trigger to S3 upload events.

Write a Lambda that will download this file's contents and parse it.

Then as parsing, you can send events to SNS, MSK, or Kinesis, or write to Athena, RDS, other S3 files, etc...

Sure, Glue might piece some of these together, but you dont "need" it for simple ETL workloads.

Related