How to stub new KafkaSink on integration test

Viewed 175

Flink 1.14 marks FlinkKafkaProducer deprecated and replaces it with the new KafkaSink class. I'm trying to put a new approach into MiniClusterWithClientResource integration tests similar as described here: Testing Flink jobs.

Unfortunately, I'm having issues stubbing the new KafkaSink class as it inherits from an awkward Sink interface with Flink internal generic types. Also KafkaSink class itself exposes only a package-private constructor - therefore it's impossible to inherit from it from the arbitrary java package and overload its behavior.

I managed to overcome the issue by inheriting from KafkaSink class in my own apache.flink.connector.kafka.sink package, however, there must be a better way of handling the testing or the new API is not really test friendly.

package org.apache.flink.connector.kafka.sink;

import org.apache.flink.api.connector.sink.Committer;
import org.apache.flink.api.connector.sink.GlobalCommitter;
import org.apache.flink.api.connector.sink.SinkWriter;
import org.apache.flink.core.io.SimpleVersionedSerializer;

import java.util.List;
import java.util.Optional;

public abstract class TestKafkaSink<T> extends KafkaSink<T> {
    public abstract void writeValue(T value);
    public abstract void clear();

    public TestKafkaSink() {
        super(null, null, null, null);
    }

    @Override
    public SinkWriter<T, KafkaCommittable, KafkaWriterState> createWriter(InitContext context, List<KafkaWriterState> states) {
        return new SinkWriter<>() {
            @Override
            public void write(T t, Context context) {
                writeValue(t);
            }

            @Override
            public List<KafkaCommittable> prepareCommit(boolean b) {
                return null;
            }

            @Override
            public void close() {

            }
        };
    }

    @Override
    public Optional<Committer<KafkaCommittable>> createCommitter() {
        return Optional.empty();
    }

    @Override
    public Optional<GlobalCommitter<KafkaCommittable, Void>> createGlobalCommitter() {
        return Optional.empty();
    }

    @Override
    public Optional<SimpleVersionedSerializer<KafkaCommittable>> getCommittableSerializer() {
        return Optional.empty();
    }

    @Override
    public Optional<SimpleVersionedSerializer<Void>> getGlobalCommittableSerializer() {
        return Optional.empty();
    }

    @Override
    public Optional<SimpleVersionedSerializer<KafkaWriterState>> getWriterStateSerializer() {
        return Optional.empty();
    }
}
1 Answers

The approach taken in Flink's integration tests for the KafkaSink is to bring up a containerized instance of Kafka, rather than mocking it. For example, you could look at https://github.com/apache/flink/blob/release-1.14/flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/sink/KafkaSinkITCase.java.

Not sure how helpful this will prove to be in this case, but in general I find looking at Flink's own tests is a good strategy for figuring out how the developers intended the interfaces to be used.

Related