I have a job that uses Flink's side output capability to write data to different kafka sinks. The side output gets data written to it when ran in the IDE but not when on the Flink cluster. Any idea why?
Here is a sample of the code:
final OutputTag<SideOutputObject> sideOutputTag = new OutputTag<SideOutputObject>("side-output-tag"){};
SingleOutputStreamOperator<String> processedDataStream = outputStream
.process(new ProcessAndSortBinaryData(sideOutputTag))
.startNewChain()
.name("processedDataStream")
.uid("processedDataStream");
DataStream<String> sideOutputObjectStream = processedDataStream.getSideOutput(sideOutputTag)
.flatMap(new FlatMapFunction<SideOutputObject, String>() {
@Override
public void flatMap(SideOutputObject sideOutputObject, Collector<String> collector) throws Exception {
System.out.println("sideOutputObject in side output flat map!");
collector.collect(sideOutputObject.toString());
}
})
.startNewChain()
.name("sideOutputStream")
.uid("sideOutputStream");
sideOutputObjectStream.addSink(new FlinkKafkaProducer010<>(
"sideOutputKafkaTopic",
new SimpleStringSchema(),
kafkaSinkProperties)
).name("sideOutput-KafkaSink")
.uid("sideOutput-KafkaSink");
The flatmap never shows that it has received records in the cluster GUI and the System.out.println message is never written out to stdout either.
Any help would be greatly appreciated. Thanks in advance!