How to set OpenSearch/Elasticsearch as the destination of a Kinesis Firehose?

Viewed 565

I am trying to create Data Stream -> Firehose -> OpenSearch infrastructure using the AWS CDK v2. I was surprised to find that, although OpenSearch is a supported Firehose destination, there is nothing in the CDK to support this use case.

In my CDK Stack I have created an OpenSearch Domain, and am trying to create a Kinesis Firehose DeliveryStream with that domain as the destination. However, kinesisfirehose-destinations package seems to only have a ready-to-use destination for S3 buckets, so there is no obvious way to do this easily using only the constructs supplied by the aws-cdk, not even using the alpha packages.

I think I should be able to write an OpenSearch destination construct by implementing IDestination. I have tried the following simplistic implementation:

import {Construct} from "constructs"
import * as firehose from "@aws-cdk/aws-kinesisfirehose-alpha"
import {aws_opensearchservice as opensearch} from "aws-cdk-lib"

export class OpenSearchDomainDestination implements firehose.IDestination {
  private readonly dest: opensearch.Domain

  constructor(dest: opensearch.Domain) {
    this.dest = dest
  }

  bind(scope: Construct, options: firehose.DestinationBindOptions): firehose.DestinationConfig {
    return {dependables: [this.dest]}
  }
}

then I can use it like so,

export class MyStack extends Stack {
  ...
  private createFirehose(input: kinesis.Stream, output: opensearch.Domain) {

    const destination = new OpenSearchDomainDestination(output)
    const deliveryStream = new firehose.DeliveryStream(this, "FirehoseDeliveryStream", {
      destinations:       [destination],
      sourceStream:       input,
    })

    input.grantRead(deliveryStream)
    output.grantWrite(deliveryStream)
  }
}

This will compile and cdk synth runs just fine. However, I get the following error when running cdk deploy:

CREATE_FAILED | AWS::KinesisFirehose::DeliveryStream | ... Resource handler returned message: "Exactly one destination configuration is supported for a Firehose

I'm not sure I understand this message but it seems to imply that it will reject outright everything except the one provided S3 bucket destination.


So, my titular question could be answered by the answer to either of these two questions:

  1. How are you supposed to implement bind in IDestination?
  2. Are there any complete working examples of creating a Firehose to OpenSearch using the non-alpha L1 constructs?

(FYI I have also asked this question on the AWS forum but have not yet received an answer.)

1 Answers

Other destinations (at the moment) than S3 are not supported by the L2 constructs. This is described at https://docs.aws.amazon.com/cdk/api/v1/docs/aws-kinesisfirehose-readme.html

In such cases, I go to the source code to see what can be done. See https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-kinesisfirehose/lib/destination.ts . There is no easy way how to inject other destination than S3 since the DestinationConfig does not support it. You can see at https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-kinesisfirehose-destinations/lib/s3-bucket.ts how the config for S3 is crafted. And you can see how that config is used to translate to L1 construct CfnDeliveryStream at https://github.com/aws/aws-cdk/blob/f82d96bfed427f8e49910ac7c77004765b2f5f6c/packages/%40aws-cdk/aws-kinesisfirehose/lib/delivery-stream.ts#L364

Probably easiest way at the moment is to write down your L1 constructs to define destination as OpenSearch.

Related