How to write and read json objects to a file in google cloud in a for loop using Python

user_1448396 0
Viewed 4415

I am trying to write a list of json objects to a file in google cloud using python. I am able to write a single object in the file. But it is not working when I try to write it in a for loop.

Here is the code which works for a single object but does not work when i write iteratively


from google.cloud import storage
import json

bucket_name = 'gcs_bucket_user'
bucket = storage.Client().get_bucket(bucket_name)

for i in range(0,5):
    json_object = {'i': 'i'}
    blob = bucket.blob('text.json')
    blob.upload_from_string(data=json.dumps(json_object),content_type='application/json')

Expected Output
{'0':'0'}
{'1':'1'}
{'2':2}
and so on

But this is not appending objects in the json file. It is overwriting them.

Also what is the way to iteratively read json objects from such a file in google cloud

google-cloud-storage google-cloud-platform python json
2 Answers

I am not familiar with the specific details regarding cloud storage; however it looks like you are overwriting the file with every loop.

First of all, json_object = {'i': 'i'} has no effect for every loop, as you need to destinate a variable.

Second, I will try to illustrate with a code example.

from google.cloud import storage
import json

bucket_name = 'gcs_bucket_user'
bucket = storage.Client().get_bucket(bucket_name)
# define a dummy dict
some_json_object = {'foo': list()}

for i in range(0, 5):
    some_json_object['foo'].append(i)

blob = bucket.blob('text.json')
# take the upload outside of the for-loop otherwise you keep overwriting the whole file
blob.upload_from_string(data=json.dumps(some_json_object),content_type='application/json')

Take the file upload outside of the loop, and bulk-append all your data to the file.

In your case you can simulate the bulk upload from the list by stitching new lines to the file with something similar to: "\n".join(['I', 'would', 'expect', 'multiple', 'lines']) or use a native method for updating (if available).

user_112710480

As for google cloud storage docs:

Objects are immutable, which means that an uploaded object cannot change throughout its storage lifetime.

So if you want to append to a object then the only way are:

  1. You can merge it before upload to Google Cloud Storage
  2. Or with each upload, you will have to download the content that already existed, merge it with the new data in your local and then upload it back.
user_63539960
Related