I have a kafka topic with several event types.
My events value look like:
type|name|value1|value2|value3
type|name|value1|value2|value3|value4
type|name|value1|value2
I used structured streaming to readstream kafka. I want to store stream data to dataframe. The result dataframe has schema which is including columns in all schemas. Can I store event records of each type in a dataframe then select from these dataframes and union them in order to result dataframe ?