UTF8-mb4 in Kafka

user_1578872 0
Viewed 123

I have a UTF8-mb4 char in mongo and have a java extractor which extracts data from mongo and put into kafka. When the data goes to Kafka the special char has been replaced with \u...

Sample text:- "\uDBFF\uDC15COMPANY"

I have another Java program which reads from one kafka topic and put it into another kafka topic after some operation. When the data is read from the actual topic, the \u... is been decoded to an actual special char and when the data is pushed to the target topic, it is is like some junk char. How to put the data back to the target topic as \u ...

The same message in the target topic is like,

"COMPANY"

Note:-

The message has lots of data(JSON data) and there could be a special char in any json value.

While reading from the source topic,

For consumer to consume from the source topic,

key.deserializer = "org.apache.kafka.common.serialization.ByteArrayDeserializer"
value.deserializer = "org.apache.kafka.common.serialization.ByteArrayDeserializer"

For produce to produce in the target topic,

key.serializer = "org.apache.kafka.common.serialization.ByteArraySerializer"
value.serializer = "org.apache.kafka.common.serialization.ByteArraySerializer"
utf8mb4 apache-kafka utf-8 java
2 Answers

Since you're using ByteArraySerializer to preserve the data exactly as written rather than StringSerializer, for example, which defaults to utf8 encoding, then you're going to potentially get those database control characters, which cannot be displayed as ASCII, so you end up with instead.

The same message in the target topic is like,

"COMPANY"

Unclear what you're using to view this data, but perhaps the issue exists in the encoding of that program, not Kafka itself, or your choice of serializer

user_23086830

You can try other serializers. I think this would be usefulmongodb

user_158351620
Related