Parameterized Redshift COPY from S3 via redshift-data API

Viewed 660

I am issuing a COPY command to connect to S3 and load data into Redshift. I am attempting to do so using the Redshift-Data API, which states that it supports parameterized queries. This is what I need. However upon attempting even the simplest example it appears that the Redshift-Data API is in fact NOT capable of issuing a COPY command with parameters, however it is happy to run other queries through the API. You can reproduce the following in whichever client you use, I tested locally via the cli but I get the same results in the Python boto3 client (which ultimately I would like to use).

Minimal reproducible example

Have a file in S3:

echo 'id,foo,is_test
1,foo,true
2,bar,true
3,baz,false' > testfile.csv
aws s3 cp ./testfile.csv s3://mybucket/testfile.csv

Try to run the copy command via the CLI (or Python, whatever you like)

###
# DOES NOT WORK
export MYFILE='s3://mybucket/testfile.csv'
aws redshift-data execute-statement \
    --database dev \
    --cluster-id redshift-clister-id \
    --secret-arn 'arn:aws:secret....' \
    --region us-east-1 \
    --sql "COPY public.table FROM :s3_uri IAM_ROLE 'my-iam-role' REGION 'us-east-1' IGNOREHEADER 1 CSV;" \
    --parameters "[{\"name\": \"s3_uri\", \"value\": \"${MYFILE}\"}]"

###
# ALSO DOES NOT WORK
export MYFILE="'s3://mybucket/testfile.csv'"
aws redshift-data execute-statement \
    --database dev \
    --cluster-id redshift-clister-id \
    --secret-arn 'arn:aws:secret....' \
    --region us-east-1 \
    --sql "COPY public.table FROM :s3_uri IAM_ROLE 'my-iam-role' REGION 'us-east-1' IGNOREHEADER 1 CSV;" \
    --parameters "[{\"name\": \"s3_uri\", \"value\": \"${MYFILE}\"}]"

###
# WORKS
export MYFILE="'s3://mybucket/testfile.csv'"
aws redshift-data execute-statement \
    --database dev \
    --cluster-id redshift-clister-id \
    --secret-arn 'arn:aws:secret....' \
    --region us-east-1 \
    --sql "SELECT COUNT(*) FROM schema.table WHERE column = :s3_uri;" \
    --parameters "[{\"name\": \"s3_uri\", \"value\": \"${MYFILE}\"}]"

###
# ALSO "WORKS" but not for my use case
aws redshift-data execute-statement \
    --database dev \
    --cluster-id redshift-clister-id \
    --secret-arn 'arn:aws:secret....' \
    --region us-east-1 \
    --sql "COPY public.table FROM 's3://mybucket/testfile.csv' IAM_ROLE 'my-iam-role' REGION 'us-east-1' IGNOREHEADER 1 CSV;" \

The result from which always gives me something to the effect of:

"Error": "ERROR: syntax error at or near \"$1\"\n  Position: ##",

Which is always pointing to the position of the start of the S3 URI variable. When I remove the parameters entirely and hard code the value, the COPY operation succeeds just fine. I have tried wrapping the parameter in single quotes and also tried the exact same parameter in a SELECT query against the API. The SELECT query works easily with no hassle, just not the COPY.

So ultimately I would like to use the parameters to more safely run the query, because I will not know the s3_uri ahead of time. I can certainly go the route of interpolating the string into the query, but at that point I have a SQL injection vector which I do not have control over (the s3 object names). I verified that you can exploit this with a given object name as s3 object names allow whitespace and any character you need in order to create an injection scenario.

I am finding the same strange behavior present in the redshift_connector where the parameters work on SELECT but not on COPY. Any link to documentation or example that states either this is not supported or shows how to do this would be great, I just cannot find it.

1 Answers

I suspect you are just missing the single quotes in the passed value. Try

export MYFILE="'s3://mybucket/testfile.csv'"

Your shell is stripping the single quotes in your example

Related