I have a bucket/folder into which a lot for files are coming in every minutes. How can I read only the new files based on file timestamp.
eg: list all files with timestamp > my_timestamp
3 Answers
You could use some bash-fu:
gsutil ls -l gs://<your-bucket-name> | sort -k2n | tail -n1 | awk 'END {$1=$2=""; sub(/^[ \t]+/, ""); print }'
breaking that down:
# grab detailed list of objects in bucket
gsutil ls -l gs://your-bucket-name
# sort by number on the date field
sort -k2n
# grab the last row returned
tail -n1
# delete first two cols (size and date) and ltrim to remove whitespace
awk 'END {$1=$2=""; sub(/^[ \t]+/, ""); print }'`
Tested with Google Cloud SDK v186.0.0, gsutil v4.28
If you are interested in new files or we can say in another words the files which are not present in your destination bucket then alternatively you can use gsutil -n option as it copies only those files which are not present in destination bucket.
From documentation https://cloud.google.com/storage/docs/gsutil/commands/cp?hl=ru
No-clobber. When specified, existing files or objects at the destination will not be overwritten. Any items that are skipped by this option will be reported as being skipped. This option will perform an additional GET request to check if an item exists before attempting to upload the data. This will save retransmitting data, but the additional HTTP requests may make small object transfers slower and more expensive.
cons with this approach is, it makes a check request for every file present in your source bucket
Related