How to configure kafka s3 sink connector for json using its fields AND time based partitioning?

user_10227460 0
Viewed 565

I have a json coming in like this: 

{
    "app" : "hw",
    "content" : "hello world",
    "time" : "2018-05-06 12:53:04"
}

I wish to push to S3 in the following file format:

/upper-directory/$jsonfield1/$jsonfield2/$date/$HH

I know I can achieve:

/upper-directory/$date/$HH

with TimeBasedPartitioner and Topic.dir, but how do I put in the 2 json fields as well?

confluent-platform apache-kafka-connect apache-kafka amazon-s3
1 Answers

You need to write your own Partitioner to achieve a combination of TimeBased and Field Partitioners

That means make a new Java project, look at the source code for a reference point, build a JAR out of the project, and then copy the jar into kafka-connect-storage-common on all servers running Kafka Connect, which is picked up by the S3 connector. After you've copy the JAR, you will need to reboot the Connect process.

Note: there's already a PR that is trying to add this - https://github.com/confluentinc/kafka-connect-storage-common/pull/73/files

user_23086830
Related