How to write custom SMT in Kafka Source Connector to obfuscate private data

user_4258834 0
Viewed 523

Suppose that I want to set up a Kafka source connector to pump some data from my DB into my Kafka. Now, suppose that due to security reasons I want to obfuscate (say, apply a hash function) to some of the columns in my DB. Is there a standard way to to that with Kafka Connect, or do I have to first create private topics and then write some custom code to create public versions of those topics (with hashed values for sensitive data)?

I know that there's such thing as transformations (MaskField, etc.), but none of them seems to solve my problem. For example, MaskField does something like +1-234-567-8987 -> "--****", which I do not want to happen, because I still want to have distinct entries for different input values.

apache-kafka-connect apache-kafka
1 Answers

Kafka Connect makes it very convenient to write your own single message transformation (SMT). If none of the default transformations serve your purpose you can build a custom transformator.

This blog explains how to build your own, together with a full example provided on GitHub:

You need to implement Transformation<R> and override the apply() method with your custom algorithm to handle personal related data.

When you compile that code as a JAR and make it available to Kafka Connect via the plugin.path specified in the Connect worker’s properties file, then the transforms can be configured in the connector properties.

user_122089100
Related