Issue accessing S3 from Tensorflow

user_10626567 0
Viewed 1142

With the following config.

os.environ['AWS_ACCESS_KEY_ID'] = 'xxxxxx'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'xxxxxxxx'
os.environ['AWS_REGION'] = 'us-west-2'
os.environ['S3_ENDPOINT'] = 's3-us-west-2.amazonaws.com'
os.environ['S3_USE_HTTPS'] = '1'                          
os.environ['S3_VERIFY_SSL'] = '1' 

print(file_io.stat('s3://abcd/def.txt'))

I get the error

/usr/local/lib/python3.6/dist-packages/tensorflow/python/lib/io/file_io.py in stat(filename) 556 with errors.raise_exception_on_not_ok_status() as status: 557 pywrap_tensorflow.Stat(compat.as_bytes(filename), file_statistics, status) --> 558 return file_statistics 559 560

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg) 526 None, None, 527 compat.as_text(c_api.TF_Message(self.status.status)), --> 528 c_api.TF_GetCode(self.status.status))
529 # Delete the underlying status object from memory otherwise it stays alive
530 # as there is a reference to status from this from the traceback due to

NotFoundError: Object s3://abcd/def.txt does not exist

Note this file does exist.

I also get the following error on a write and close.

UnknownError: PermanentRedirect: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

What more is needed to fix this?

tensorflow amazon-s3
2 Answers

This is how my config looks like:

import os
os.environ['AWS_REGION'] = 'us-west-2'
os.environ['S3_ENDPOINT'] = 'https://s3-us-west-2.amazonaws.com'
os.environ['S3_VERIFY_SSL'] = '0'

I think you have to change from

os.environ['S3_ENDPOINT'] = 's3-us-west-2.amazonaws.com'

to

os.environ['S3_ENDPOINT'] = 'https://s3-us-west-2.amazonaws.com'

Here is a link for your reference.

user_83673070

you can use endpoint like mybucket.s3-us-west-2.amazonaws.com

then use s3://pathtofile to access

user_106065340
Related