Merge Kafka record Values from single stream

user_5567528 0
Viewed 136

I have a topic which receives JSON records with possible partial data. I want to merge this data, so i try to gather as much information as possible within the final data record.

 t1: { id: '1234', attribute1: 'foo' }
 t2: { id: '1234', attribute2: 'bar' }

desired stream after merging record values:

 t1: { id: '1234', attribute1: 'foo' }
 t2: { id: '1234', attribute1: 'bar', attribute2: 'bar' }

To achieve this I tried:

 //key of the topic is id
 KStream<String, MyObject> input = ...
 return input.groupByKey().reduce((current, newEvent) -> return newEvent.merge(current)).toStream();

but that yields only a single entry, as groupy/reduce produces a KTable. Is there a possibility to achieve this?

Edit: The stream definition was correct, it appears that reduce by default does not send all messages downstream, but caches them before doing so. To disable this behaviour the configuration property:

 cache.max.bytes.buffering: 0

has to be set.

spring-kafka apache-kafka-streams java
1 Answers

Try this

public class MergeStreams {

    public Topology buildTopology(Properties allProps) {
        final StreamsBuilder builder = new StreamsBuilder();

        final String rockTopic = allProps.getProperty("input.rock.topic.name");
        final String classicalTopic = allProps.getProperty("input.classical.topic.name");
        final String allGenresTopic = allProps.getProperty("output.topic.name");

        KStream<String, SongEvent> rockSongs = builder.stream(rockTopic);
        KStream<String, SongEvent> classicalSongs = builder.stream(classicalTopic);
        KStream<String, SongEvent> allSongs = rockSongs.merge(classicalSongs);

        allSongs.to(allGenresTopic);
        return builder.build();
    }

    public void createTopics(Properties allProps) {
        AdminClient client = AdminClient.create(allProps);

        List<NewTopic> topics = new ArrayList<>();

        topics.add(new NewTopic(
                allProps.getProperty("input.rock.topic.name"),
                Integer.parseInt(allProps.getProperty("input.rock.topic.partitions")),
                Short.parseShort(allProps.getProperty("input.rock.topic.replication.factor"))));

        topics.add(new NewTopic(
                allProps.getProperty("input.classical.topic.name"),
                Integer.parseInt(allProps.getProperty("input.classical.topic.partitions")),
                Short.parseShort(allProps.getProperty("input.classical.topic.replication.factor"))));

        topics.add(new NewTopic(
                allProps.getProperty("output.topic.name"),
                Integer.parseInt(allProps.getProperty("output.topic.partitions")),
                Short.parseShort(allProps.getProperty("output.topic.replication.factor"))));

        client.createTopics(topics);
        client.close();
    }

    public Properties loadEnvProperties(String fileName) throws IOException {
        Properties allProps = new Properties();
        FileInputStream input = new FileInputStream(fileName);
        allProps.load(input);
        input.close();

        return allProps;
    }

    public static void main(String[] args) throws Exception {
        if (args.length < 1) {
            throw new IllegalArgumentException("This program takes one argument: the path to an environment configuration file.");
        }

        MergeStreams ms = new MergeStreams();
        Properties allProps = ms.loadEnvProperties(args[0]);
        allProps.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        allProps.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, SpecificAvroSerde.class);
        allProps.put(SCHEMA_REGISTRY_URL_CONFIG, allProps.getProperty("schema.registry.url"));
        Topology topology = ms.buildTopology(allProps);

        ms.createTopics(allProps);

        final KafkaStreams streams = new KafkaStreams(topology, allProps);
        final CountDownLatch latch = new CountDownLatch(1);

        // Attach shutdown handler to catch Control-C.
        Runtime.getRuntime().addShutdownHook(new Thread("streams-shutdown-hook") {
            @Override
            public void run() {
                streams.close(Duration.ofSeconds(5));
                latch.countDown();
            }
        });

        try {
            streams.start();
            latch.await();
        } catch (Throwable e) {
            System.exit(1);
        }
        System.exit(0);
    }
}

Need more details, clickhere

user_159417900
Related