I've found this nice article on how to directly stream data from Google Storage to tf.data. This is super handy if your compute tier has limited storage (like on KNative in my case) and network bandwidth is sufficient (and free of charge anyway).
tfds.load(..., try_gcs=True)
Unfortunately, my data resides in a non Google bucket and it isn't documented for other Cloud Object Store systems.
Does anybody know if it also works in non GS environments?
I'm not sure how this is implemented in the library, but it should be possible to access other object store systems in a similar way.
You might need to extend the current mechanism to use a more generic API like the S3 API (most object stores have this as a compatibility layer). If you do need to do this, I'd recommend contributing it back upstream, as it seems like a generally-useful capability when either storage space is tight or when fast startup is desired.
