I'm trying to integrate a Google Cloud Dataflow pipeline with Google Cloud Pub/Sub Notifications for Google Cloud Storage.
The idea is start processing a file as soon it is created.
The messages are being published and with PubsubIO.readMessagesWithAttributes() source I manage to extract the file URI:
Pipeline p = Pipeline.create(options);
PCollection<String> uris = p.apply(PubsubIO.readMessagesWithAttributes()
.withTimestampAttribute(PUBSUB_TIMESTAMP_LABEL_KEY)
.fromSubscription(options.getPubsubSubscription()))
.apply(MapElements
.into(TypeDescriptors.strings())
.via((PubsubMessage msg) -> {
String bucket = msg.getAttribute("bucketId");
String object = msg.getAttribute("objectId");
GcsPath uri = GcsPath.fromComponents(bucket, object);
return uri.toString();
}));
Which PTransform can be used to start reading/processing each file in the uris PCollection?