Quarkus - KStream and KTable join does not output messages

Viewed 26

I am building a project modeled on this project. The key difference is, I want to output, conditionally, a message using the messages from the joined topics. As opposed to the example project, where an aggregation is performed. I am struggling to use Serde for JSON messages and so, I have simplified the message structure as follows.

  • t1 (KStream) - a plain text value.
  • t2 (KTable) - a plain text value separated by a ;.
  • t3 (KStream) - a CSV string.

I am publishing messages using kafkacat with the -k option to set a key e.g. k1. The problem I am facing is: I don't see any output in t3.

This is my TopologyProducer.java.

@Produces
  public Topology buildTopology() {
    StreamsBuilder builder = new StreamsBuilder();

    ObjectMapperSerde<stream1> stream1 = new ObjectMapperSerde<>(stream1.class);
    ObjectMapperSerde<topic1> topic1 = new ObjectMapperSerde<>(topic1.class);
    ObjectMapperSerde<output1> output1 = new ObjectMapperSerde<>(output1.class);

    GlobalKTable<String, topic1> topic1 = builder.globalTable(
        t2,
        Consumed.with(Serdes.String(), topic1));

    builder.stream(t1,
        Consumed.with(Serdes.String(), stream1))
        .join(t2,
            (paramName, paramValue) -> paramName,
            (paramValue, paramLimits) -> {
              // Add some logic to return conditionally
              return new output1("paramName", 0.0, 0.0, true);
            })
        .to(t3,
            Produced.with(Serdes.String(), output1));
    return builder.build();
  }
}
1 Answers

The Java version I had in my Dockerfile was wrong.

When I inspected the container logs, I saw an error about the difference in version of Java used to compile (higher) vs running (lower). I chose the simpler of two i.e. used a more recent version of Java to run the application (than, adjusting the Java version for local mvn build). The error can be traced to the following instruction as documented here.

The Dockerfile created by Quarkus by default needs one adjustment for the aggregator application in order to run the Kafka Streams pipeline. To do so, edit the file aggregator/src/main/docker/Dockerfile.jvm and replace the line FROM fabric8/java-alpine-openjdk8-jre with FROM fabric8/java-centos-openjdk8-jdk.

I edited my Dockerfile to use FROM registry.access.redhat.com/ubi8/openjdk-17:1.11 and have the application running.

Related