How to store numerical EC2 computation results - serialize a Julia object and a Julia DataFrame in an AWS S3 bucket?

user_9957710 0
Viewed 98

I am running my large scale Julia computations using an Amazon EC2 instance. I would like to save the state of computations (Julia object) as well the final results (DataFrame) to an S3 bucket.

Suppose my data structure is:

struct SampleData
    a::Int
    b::String
end

and I have an object:

d = SampleData(1,"sss")

I have already followed the tutorial by Amazon https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/ and have created an IAM role with a appropriate access rights to S3 and this role is already attached to my EC2 instance.

What I now need to do to save the d object from Julia directly to S3?

On the other hand my computation results are a DataFrame. Suppose my data frame looks as follows:

using DataFrames, CSV
df = DataFrame(a=1:3,b='a':'c',c=1.5:1:3.5)

What is the good way to write it directly to S3?

julia amazon-web-services amazon-ec2 amazon-s3
1 Answers

The AWS3.jl can work with byte arrays and be used to serialize such data. Hence one can run the code:

using AWS, AWSS3, Serialization

aws = global_aws_config(; region="us-east-1") #choose the appropiate region

b = IOBuffer()
serialize(b, d)

s3_put(aws, "your-s3-bucket-name","myfile.jld", take!(b))

Once written this data can be retrieved any time to Julia by using the following code:

ddat = s3_get(aws, "your-s3-bucket-name","myfile.jld")
d_copy = deserialize(IOBuffer(ddat))

A DataFrame can be written to S3 with the following code:

b = IOBuffer()
CSV.write(b,df)
s3_put(aws, "your-s3-bucket-name","mydf.csv", take!(b))

And this can be read back from S3 simply by:

df_copy = CSV.read(IOBuffer(s3_get(aws, "your-s3-bucket-name","mydf.csv")), DataFrame)
user_99577100
Related