Data written to side output in IDE but not on cluster

Viewed 509

I have a job that uses Flink's side output capability to write data to different kafka sinks. The side output gets data written to it when ran in the IDE but not when on the Flink cluster. Any idea why?

Here is a sample of the code:

final OutputTag<SideOutputObject> sideOutputTag = new OutputTag<SideOutputObject>("side-output-tag"){};

SingleOutputStreamOperator<String> processedDataStream = outputStream
                .process(new ProcessAndSortBinaryData(sideOutputTag))
                .startNewChain()
                .name("processedDataStream")
                .uid("processedDataStream");

DataStream<String> sideOutputObjectStream = processedDataStream.getSideOutput(sideOutputTag)
                .flatMap(new FlatMapFunction<SideOutputObject, String>() {
                    @Override
                    public void flatMap(SideOutputObject sideOutputObject, Collector<String> collector) throws Exception {
                        System.out.println("sideOutputObject in side output flat map!");
                        collector.collect(sideOutputObject.toString());
                    }
                })
                .startNewChain()
                .name("sideOutputStream")
                .uid("sideOutputStream");


sideOutputObjectStream.addSink(new FlinkKafkaProducer010<>(
                    "sideOutputKafkaTopic",
                    new SimpleStringSchema(),
                    kafkaSinkProperties)
            ).name("sideOutput-KafkaSink")
                    .uid("sideOutput-KafkaSink");

The flatmap never shows that it has received records in the cluster GUI and the System.out.println message is never written out to stdout either.

Any help would be greatly appreciated. Thanks in advance!

1 Answers
Related