Flink TypeInfo Java Generics

Viewed 217

I want my Flink app to deserialize data from a Kafka topic either to Flink stream of ConsumerRecord<byte[], byte[]> or a Flink stream of Tuple2<byte[], byte[]> using the Flink org.apache.flink.api.java.tuple.Tuple2 class.

With both options I can't get a getProducedType implementation that compiles.

public class KafkaNOOPDeserialization implements KafkaDeserializationSchema<ConsumerRecord<byte[], byte[]>> {
    @Override
    public boolean isEndOfStream(ConsumerRecord<byte[], byte[]> nextElement) {
        return false;
    }

    @Override
    public ConsumerRecord<byte[], byte[]> deserialize(ConsumerRecord<byte[], byte[]> record) throws Exception {
        return record;
    }

    @Override
    public TypeInformation<ConsumerRecord<byte[], byte[]>> getProducedType() {
        // This provides TypeInformation<ConsumerRecord>
        // It needs TypeInformation<ConsumerRecord<byte[], byte[]>>
        var typeInfo = TypeExtractor.getForClass(ConsumerRecord.class);
        // This hard cast won't compile
        return (TypeInformation<ConsumerRecord<byte[], byte[]>>) typeInfo;
    }
}
public class KafkaByteArrayTupleDeserializer implements KafkaDeserializationSchema<Tuple2<byte[], byte[]>> {
    @Override
    public boolean isEndOfStream(Tuple2<byte[], byte[]> nextElement) {
        return false;
    }

    @Override
    public Tuple2<byte[], byte[]> deserialize(ConsumerRecord<byte[], byte[]> record) throws Exception {
        return new Tuple2(record.key(), record.value());
    }

    @Override
    public TypeInformation<Tuple2<byte[], byte[]>> getProducedType() {
        // This provides TupleTypeInfo<Tuple>
        // It needs TupleTypeInfo<Tuple2<byte[], byte[]>>
        var typeInfo = TupleTypeInfo.getBasicAndBasicValueTupleTypeInfo(
                byte[].class, byte[].class);
        // Hard cast won't compile
        return (TypeInformation<Tuple2<byte[], byte[]>>) typeInfo;
    }
}
1 Answers

Flink provides the org.apache.flink.api.common.typeinfo.TypeHint class that helps with this. (documentation)

For instance in your second example instead of:

return (TypeInformation<Tuple2<byte[], byte[]>>) typeInfo;

You could write:

return new TypeHint<Tuple2<byte[], byte[]>>(){}.getTypeInfo();
Related