KSQLDB: parent-child join to elasticsearch

user3738017 0
Viewed 38

I want to create a parent-child relation from the tables I have in KSQLDB to elasticsearch.

In elasticsearch mapping I just define the the join field:

    "mappings": {
        "properties": {
            "join_field": {
              "type": "join",
              "relations": {
                "parent": "child"
              }
            },
            ...

For parent in ksqldb i just add a "join_field" in the select in ksqldb:

  SELECT
    'parent' AS join_field, \
    ...

But how do I create the child? Elasticsearch is expecting this format:

"join_field": {
    "name": "child", 
    "parent": "1" 
}

I use "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector" to push data from ksqldb to elasticsearch.

ksqldb elasticsearch
1 Answers

Its not implemented yet. Hopefully soon. https://github.com/confluentinc/kafka-connect-elasticsearch/pull/558

While waiting for an official release. It is possible to build the version that support routing with this commit https://github.com/confluentinc/kafka-connect-elasticsearch/tree/2e25ed8329fad1dc15ba5711e7ed32e0c270dd5d

This config is needed when creating the sink: "routing.field.name" = "PARENTID"

In KSQLDB select use struct: STRUCT(`name` := 'child', `parent` := parentid) AS join_field, \

user37380170
Related