Google AI platform can't write to Cloud Storage

Question

Running a tensorflow-cloud job on Google AI Platform, the entrypoint of the job is the following:

import tensorflow as tf

filename = r'gs://my_bucket_name/hello.txt'
with tf.io.gfile.GFile(filename, mode='w') as w:
  w.write("Hello, world!")

with tf.io.gfile.GFile(filename, mode='r') as r:
  print(r.read())

The job completed successfully, in the logs it prints "hello world".

The bucket and the job are both in the same region.

But I can't find the file in Cloud Storage. It is not there. I ran some other tests, where I did tf.io.gfile.listdir then wrote a new file and again tf.io.gfile.listdir, I printed the before and after, it seems that a file was added but when I open cloud storage, I can't find it there. Also was able to read files from storage.

I'm not getting any permissions errors, and as the official docs say, AI Platform already has the permission to read/write to Cloud Storage.

Here is my main.py file:

import tensorflow_cloud as tfc

tfc.run(
   entry_point="run_me.py",
   requirements_txt="requirements.txt",
   chief_config=tfc.COMMON_MACHINE_CONFIGS['CPU'],
   docker_config=tfc.DockerConfig(
      image_build_bucket="test_ai_storage"),
)

This is the most minimal version where I can reproduce the problem.

user_4375999 · Answer

Cloud Storage is not a file system. Having this in mind you can perform uploads, downloads or Deletion operations in a bucket.

What you are trying to do is to open a file and write into it. What you should do is to create your file locally and then upload it to your desired bucket.

from google.cloud import storage


def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # bucket_name = "your-bucket-name"
    # source_file_name = "local/path/to/file"
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_filename(source_file_name)

    print(
        "File {} uploaded to {}.".format(
            source_file_name, destination_blob_name
        )
    )

Google AI platform can't write to Cloud Storage

1 Answers