I need to push the data of dataFrame's column value to kafka and not the column name, how I will do that if I have data of column like [{"a":"1"}, {"b":"2"}]
1 Answers
In order to write a Dataframe Kafka, you're required to have at least one column named value. The column name, literal string "value" is not written to any content of the Kafka record.
Unclear how you got a column named [{"a":"1"}, {"b":"2"}], but you should alias("value") it as part of a select query, then you'll have a Kafka record with a null key and value payload of that JSON array. If you want two events of {"a":"1"} and {"b":"2"}, then you should use an explode() function of SparkSQL first
Related