Not able to save model to gs bucket using torch.save()

user_16859286 0
Viewed 314

I am trying to save a PyTorch model to my google cloud bucket but it is always showing a "FileNotFoundError error".

I already have a gs bucket and the file path I am providing is also correct. My code is running on a GCP notebook instance.

path = "gs://bucket_name/model/model.pt"
torch.save(model,path)

It would really help me if someone tries to upload a model to gs bucket and let me know if you could or not? It would also help me if you shared the right way to put models into gs bucket using torch.save().

pytorch google-cloud-platform python
2 Answers

You have to use torch.save in a slightly different way. As written in the docs, the f parameter of torch.save can either be a simple string or os.PathLike object containing a file name, or a file-like object for which the write and flush methods are implemented. We can use the latter to save the model directly to a Google Cloud bucket:

from google.cloud import storage

storage_client = storage.Client("bucket_name")
bucket = storage_client.bucket("bucket_name")
blob = bucket.blob("model/model.pt")
with blob.open("wb", ignore_flush=True) as f:
    torch.save(obj, f)
user_135137000

try this:

import gcsfs
fs = gcsfs.GCSFileSystem(project = '<enter your gc project>')
with fs.open("gs://bucket_name/"+f'model/model.pt', 'wb') as f:
    torch.save(model, f)
user_129640240
Related