How to stream upload CSV data to Google Cloud Storage (Python)

user2999349 0
Viewed 52

I'm trying to upload data from a query to a CSV file in Google Cloud Storage. To prevent the system from running out of RAM ( the result of the query is huge ), I'm streaming the query results, and need to stream those to a CSV file in Google Cloud Storage.

The documentation on streaming to Google Cloud Storage shows an example with a file_obj, which is then uploaded, but I don't see how I can stream the data dynamically using this methodology.

E.g. what I'm looking for is something like this:

blob.start_stream_upload()
for(record in db_query_stream):
    record = ",".join(record) # turn db query record into CSV data line
    blob.write_to_stream(record)
blob.finish_stream_upload()

Where write_to_stream would add record to the upload stream to blob ( the CSV file / blob object in the Google Cloud Storage ). For reference, the db_query_stream is an iterable object that dynamically streams results according to SQLAlchemy's stream_results option.

If anyone knows and could share how to do this, would much appreciate it!

google-cloud-storage sqlalchemy stream google-cloud-platform python
1 Answers

Use google.cloud.storage.fileio.BlobWriter:

from google.cloud import Client
from google.cloud.storage.fileio import BlobWriter
storage_client = Client()
bucket = storage_client.bucket('my_bucket')
blob = bucket.blob('my_object')

writer = BlobWriter(blob)
for(record in db_query_stream):
    record = ",".join(record) # turn db query record into CSV data line
    writer.write(record)
writer.close()
userUlif0
Related