How to test a Kafka Stream app without duplicating the topology? Use of TopologyTestDriver?

Viewed 613

I think I misunderstood something with the TopologyTestDriver.

Until now, when I developed a Kafka Stream app, I connect to a real Kafka in test environment, manually create and fed in-topics, and check the content of what is in the out-topic(s).

In order to automatize the tests, I looked at the TopologyTestDriver. According to the documentation published by Confluent ( https://docs.confluent.io/current/streams/developer-guide/test-streams.html ), it seems to be what I am looking for:

[...] Normally, you run the topology using the KafkaStreams class, which connects to your broker [...]. For testing, running a broker [...] adds a lot of complexity and time.

Streams provides the TopologyTestDriver [...] replacement for the KafkaStreams class. It has no external system dependencies [...] There are hooks for verifying data sent to output topics [...].

My concern is about the location of the topology. In each example I saw (on Confluent, on Kafka website, ...), the topology is defined within the test file :

private TopologyTestDriver testDriver;
....

@Before
public void setup() {
    Topology topology = new Topology();
    topology.addSource("sourceProcessor", "input-topic");
    .......

    // setup test driver
    Properties config = new Properties();
    ....
    testDriver = new TopologyTestDriver(topology, config);

    // pre-populate store
    .....
}

@After
public void tearDown() {
    testDriver.close();
}

@Test
public void shouldFlushStoreForFirstInput() {
    testDriver.pipeInput(recordFactory.create("input-topic", "a", 1L, 9999L));
    OutputVerifier.compareKeyValue(......);
    Assert.assertNull(........);
}

How can I implement tests for my existing Kafka Streams apps? I cannot imagine I have to duplicate the code of my topology - one version in my main file and one version in my test file. It would be hard to maintain.

My existing apps have the topology defined directly in the main method. I could extract the topology design from the main by adding a static method like protected static org.apache.kafka.streams.Topology buildTopology() {...}, and I would be able to call this method from the test file.

But isn't there any cleaner solution to execute the main code from the test code?

0 Answers
Related