How to pass in AWS environmental variables to Nextflow for use in Docker container

jh_ 0
Viewed 26

I would like to run a Nextflow pipeline through a Docker container. As part of the pipeline I would like to push and pull from AWS. To accomplish this end, I need to pass in AWS credentials to the container, but I do not want to write them into the image.

Nextflow has an option to pass in environmental variables as part of the Docker scope via the envWhitelist option, however I have not been able to find an example for correct syntax when doing this.

I have tried the following syntax and get an access denied error, suggesting that I am not passing in the variables properly.

docker {
 enabled = true
 envWhitelist = "AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID"
}

I explicitly passed these variables into my environment and I can see them using printenv.

Does this syntax seem correct? Thanks for any help!

nextflow docker
1 Answers

Usually you can just keep your AWS security credentials in a file called ~/.aws/credentials:

If AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are not defined in the environment, Nextflow will attempt to retrieve credentials from your ~/.aws/credentials or ~/.aws/config files.

Alternatively, you can declare your AWS credentials in your nextflow.config (or in a separate config profile) using the aws scope:

aws {
    accessKey = '<YOUR S3 ACCESS KEY>'
    secretKey = '<YOUR S3 SECRET KEY>'
    region = '<REGION IDENTIFIER>'
}

You could also use an IAM Instance Role to provide your credentials.

userrC930
Related