Transferring files modified in the last 10 minutes from S3 to Big Query using Big Query Data Transfer Service

Viewed 192

On Big Query's Data Transfer Service Documentation for S3, it states:

To reduce the possibility of missing data, schedule your Amazon S3 transfers to occur at least 10 minutes after your files are added to the bucket.

By default, when running a transfer, the logs says that it only transfers files that have a last modified time of 10 minutes ago.

However, based on the documentation, it seems that there is a way to modify this setting. When I create a transfer (whether in console or in Java), I can't seem to find a way to make it transfer files are less than 10 minutes old. How to do it?

0 Answers
Related