Run AWS Athena’s queries with Lambda function

Question

I created a table on AWS Athena on which I can run any query without any error:

select * from mytestdb.test

The table has three columns, customer_Id, product_Id, price.

I tried to create a lambda function that run the same query for me using boto3:

import time
import boto3

DATABASE = 'mytestdb'
TABLE = 'test'

output='s3://mybucketons3/'

COLUMN = 'Customer_Id'

def lambda_handler(event, context):

    keyword = 'xyz12345'

    query = "SELECT * FROM %s.%s where %s = '%s';" % (DATABASE, TABLE, COLUMN, keyword)

    client = boto3.client('athena')

    # Execution
    response = client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={
            'Database': DATABASE
        },
        ResultConfiguration={
            'OutputLocation': output,
        }
    )


    return

However I got the following error:

Response:
{
  "errorMessage": "An error occurred (AccessDeniedException) when calling the StartQueryExecution operation: User: arn:aws:sts::076088932150:assumed-role/Test/QueryTest is not authorized to perform: athena:StartQueryExecution on resource: arn:aws:athena:us-west-2:076088932150:workgroup/primary",
  "errorType": "ClientError",

It seems sort of access issue however I am not sure why because I have both lambda and athena db with the same account.

user_1405942 · Answer

As I've mentioned in the comment, your Lambda role should contain Allow policy to interact with Athena service. I've also added full permissions for your S3 bucket. Example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1547414166585",
      "Action": [
        "athena:StartQueryExecution"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "Stmt1547414166586",
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    } 
  ]
}

user_10321537 · Answer

A.Khan's idea worked for me.

Use the AWS console to edit the Lambda's IAM role to have AmazonAthenaFullAccess and AmazonS3FullAccess policies.

user_10584650 · Answer

Providing/adding full access for a service is not the best practice. You can try restricting access to only the actions that your lambda needs to perform. Try to redeploy the IAM role with specific permissions and re-attach it to lambda function once deployed successfully. Your lambda will surely work. If it still gives access denied after adding required permissions then raise a aws support ticket from your account.

Run AWS Athena’s queries with Lambda function

3 Answers