Can Cloudformation be used to setup API Gateway to forward http payloads directly to Kinesis Datastream?

user_333276 0
Viewed 578

I am attempting to describe in a cloudformation template, a HTTP listening API gateway to post any incoming json requests through to a Kinesis data stream, with the intention of batch processing later.

When the API Gateway has a protocol of HTTP however, the integration type can only be one of AWS_PROXY, or HTTP_PROXY.

When I use AWS_PROXY, then I receive the message that I can only stream to either Lambda, or Firehose.

Am I missing something? Can I not put the incoming messages directly onto a Kinesis datastream?

Thanks.

amazon-kinesis aws-sam aws-api-gateway amazon-cloudformation amazon-web-services
2 Answers

It is certainly possible. Kinesis-PutRecord is available as a first class integration for connecting an HTTP API route to an AWS service API.

Here's an example using an AWS Serverless Application Model template.

Resources:
MyApi:
    Type: AWS::Serverless::HttpApi
    Properties:
        StageName: dev
        DefinitionBody:
            openapi: "3.0.0"
            info:
                title: !Ref AWS::StackName
            paths:
                /endpoint:
                    post:
                        responses:
                            default:
                                statusCode: 200
                        x-amazon-apigateway-integration:
                            credentials: !GetAtt ApiGatewayKinesisRole.Arn
                            integrationSubtype: Kinesis-PutRecord
                            requestParameters:
                                StreamName: !Ref MyStream
                                Data: $request.body
                                PartitionKey: $request.body.id
                            type: "aws_proxy"
                            payloadFormatVersion: "1.0"
                            connectionType: "INTERNET"
                                    

MyStream:
    Type: AWS::Kinesis::Stream
    Properties:
        ShardCount: 1

ApiGatewayKinesisRole:
  Type: "AWS::IAM::Role"
  Properties:        
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        -
          Effect: "Allow"
          Action:
            - "sts:AssumeRole"
          Principal:
            Service:
              - "apigateway.amazonaws.com"
    Policies:
      -   PolicyName: "KinesisPutRecord"
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "kinesis:PutRecord"
                  - "kinesis:DescribeStream"
                  - "kinesis:GetRecords"
                  - "kinesis:GetShardIterator"
                  - "kinesis:ListShards"
                  - "kinesis:ListStreams"
                Resource:
                  - !GetAtt MyStream.Arn
user_60033070

Yes you can. For that you have to use AWS integration type.

In fact AWS docs have a tutorial how to properly integrate API gateway with Kinesis using the AWS integration type:

It also shows how to inject messages to your stream through API gateway.

Based on the information from the tutorial, you can setup cloudformation for that. I did it before without any issues, so its certenly possible.

user_2488230
Related