Kafka Connect transforming JSON string to actual JSON

Question

I'm trying to figure out whether it's possible to transform JSON values that are stored as strings into actual JSON structures using Kafka Connect.

I tried looking for such a transformation but couldn't find one. As an example, this could be the source:

{
  "UserID":2105058535,
  "DocumentID":2105058535,
  "RandomJSON":"{\"Tags\":[{\"TagID\":1,\"TagName\":\"Java\"},{\"TagID\":2,\"TagName\":\"Kafka\"}]}"
}

And this is my goal:

{
  "UserID":2105058535,
  "DocumentID":2105058535,
  "RandomJSON":{
    "Tags":[
      {
        "TagID":1,
        "TagName":"Java"
      },
      {
        "TagID":2,
        "TagName":"Kafka"
      }
    ]
  }
}

I'm trying to make these transformations for Elasticsearch sink connector if it makes a difference.

I know I can use Logstash together with JSON filter in order to do this, but I'd like to know whether there's a way to do it using just Kafka Connect.

user_350613 · Answer

Sounds like this would be a Single Message Transform (thus applicable to any connector, not just ES), but there aren't any out of the box doing what you describe. The API is documented here.

user_8027716 · Answer

I had a similar issue but in reverse. I had the data in Json and I needed to convert some of it into a Json string representation to store it in Cassandra using the Cassandra Sink. I ended up creating a Kafka stream app that reads from the topic and then output the Json object to another topic that is read by the connector.

topic document <- read by your kafka stream with a call to mapValues or create a Jackson POJO that serializes as you want, and then write value to -> topic document.elasticsearch

user_1759289 · Answer

You can use FromJson converter. Please check this link for more details https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-json-schema/transformations/examples/FromJson.inline.html

Kafka Connect transforming JSON string to actual JSON

3 Answers