How to convert Documents read from Mongo Collection using MongoDbIO.read() directly to a table row without explicity mentioning each and every key?

user_1426451 0
Viewed 117
Read read= MongoDbIO.read().withUri(<uri>).withBucketAuto(true)
            .withDatabase(<db>).withCollection(<collection>);

PCollection<Document> lines = p.apply(read);

How to convert lines to tablerows dynamically without fetching each line and setting the table row like this:

String offer_id=document.getString("oid");
TableRow row;
row=new TableRow().set("offerId", offer_id)
apache-beam google-cloud-dataflow mongodb java
1 Answers

I don't think it's possible now since anyway MongodbIO.Read will return a PCollection<Document> that has to be converted into PCollection<TableRow>.

However, if you need it to write the result of MongodbIO.Read into BigQuery then I think you may try to use BigQueryIO.Write.withFormatFunction(...) and provide a function to convert MongoDB document into TableRow. Though, I think it will be kind of the same in terms of performance.

user_106873250
Related