how to insert data into mysql table from kafka

user_5067173 0
Viewed 639

I'm studying kafka and saw how the producer and consumer structure works, I found it very useful and simple.

Producer:
$ ./kafka-console-producer.sh --broker-list  "BootstrapBrokerStringTls" --producer.config client.properties --topic AWSKafkaTutorialTopic

Consumer:
$ ./kafka-console-consumer.sh --bootstrap-server  "BootstrapBrokerStringTls" --consumer.config client.properties --topic AWSKafkaTutorialTopic --from-beginning

But to try to do something closer to the real world, I want to do when I send a json in my products, be inserted the data in a mysql table. Does anyone have any tips on how to do this?

I saw this link, but here the consumer is watching when I enter a data in mysql, and that's not what I want. On the contrary, I want what I write in the producer to be inserted in the BD.

apache-kafka mysql
1 Answers

take a look at Kafka Connect, Sink Connector for MySQL you can look at examples here: https://github.com/usdot-fhwa-stol/cav-education/tree/develop/cav_box/docker

you also need to use mysql-connector-java to create a sink. then you can use this to register your sink:

    "name": "jdbc_source_mysql",
    "config": {
            "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
            "connection.url": "jdbc:mysql://mysql:3306/name_of_database",
            "connection.user": "root",
            "connection.password": "your_password",
            "topics": "your_topic",
            "mode":"bulk",
            "poll.interval.ms" : 3600000,
            "auto.create": "true"
    }
}```
user_62700200
Related