Load pytorch model from S3 bucket

user_12486121 0
Viewed 2179

I want to load a pytorch model (model.pt) from a S3 bucket. I wrote the following code:

from smart_open import open as smart_open
import io

load_path = "s3://serial-no-images/yolo-models/model4/model.pt"
with smart_open(load_path) as f:
    buffer = io.BytesIO(f.read())
    model.load_state_dict(torch.load(buffer))

This results in the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

One solution would be to download the model locally, but I want to avoid this and load the model directly from S3. Unfortunately, I couldn't find a good solution for that online. Can someone help me out here?

torch pytorch amazon-s3
1 Answers

According to the documentation, the following works:

from smart_open import open as smart_open
import io

load_path = "s3://serial-no-images/yolo-models/model4/model.pt"
with smart_open(load_path, 'rb') as f:
    buffer = io.BytesIO(f.read())
    model.load_state_dict(torch.load(buffer))

I have tried this before, but didn't see that I have to set 'rb' as argument.

user_124861210
Related