How should I handle fetching large shared files from S3 for containers to batch process in parallel

user_4886050 0
Viewed 407

I have some largish files in S3 - up to 40G

I want to set multiple containers to work on each file using AWS Batch - each container will take a smallish slice of the larger file and analyse it, returning it's small chunk of results to S3.

Because of the size, I don't want to retrieve an individual copy of the large file for each and every container.

Because I'm using AWS Batch, the number of underlying servers and there individual sizes is not necessarily known ( it depends on spot pricing), so pre-copying of all raster to all servers isn't a good option.

At some point a container will be tasked to process chunk of a file which is not yet on the local volume - it's easy to copy it across, the problem is, what happens when 2 or more containers find that the file isn't there and start to copy it over at the same time?

So, my question is 'What is the best pattern to use to ensure that the file each container needs is available on a shared volume on the host that that particular container is running on, without creating race conditions and container inter-dependencies?'

TIA Simon

aws-batch docker amazon-s3
1 Answers

I would architect this by assigning the chunk of the file to the Batch job when I submit the job. For example, I would assign Batch Job A to process bytes 0-1G, Batch Job B to process bytes 1G-2G and so on.

You can accomplish this by setting environment variables indicating the range when you submit the job. For example, for the first job you submit, you could set RANGE_START=0 and RANGE_END=999999999. Then inside the container, you can fetch that range of bytes from S3 and do your processing. For example, in Python, you could do:

import boto3
s3 = boto3.client("s3")
obj = s3.get_object(
    Bucket="bucket",
    Key="key",
    Range="bytes 0-999999999"
)
content = obj["Body"].read()
user_95530480
Related