How to use the package AWSS3.jl to connect to an S3 bucket running on an EC2 instance?

user_2405703 0
Viewed 288

I am trying to use Julia on an AWS computing cluster, processing data from Amazon S3 buckets.

How do I use in Julia AWSS3.jl to open a connection to an S3 bucket?

In particular I would like to know what is the best approach to configure the connectivity and subsequently use that configuration from Julia.

julia amazon-web-services amazon-s3
1 Answers

  1. Create an AWS policy with S3 access rights (in particular PutObject, GetObject etc.)

  2. Create an EC2 oriented IAM role and add the policy from the first step to that role

  3. Assign the IAM role to the EC2 instance (or perhaps configure it to use in an instance profile when creating EC2 instances for your cluster)

  4. Now you are ready to do your work in Julia. Below a simple example that serializes and deserializes any Julia object to an S3 bucket.

using AWS, AWSS3, Serialization
struct SampleData
  a::Int
  b::String
end

d=SampleData(1,"sss")
aws = global_aws_config(; region="us-east-1")
b = IOBuffer()
serialize(b, d)

s3_put(aws, "your-s3-bucket-name","myfile.bin", b.data)

ddat = s3_get(aws, "your-s3-bucket-name","myfile.bin")
d2 = deserialize(IOBuffer(ddat))

@assert d == d2

Should you have any problems with points 1-3, here is the tutorial: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/

user_99577100
Related