In my Spark Structured Streaming application, I'm doing unionByName on three Datasets:
val allRecords = ds1
.unionByName(ds2)
.unionByName(ds3)
where ds1 etc are strongly-typed Datasets. Sometimes, one Dataset is empty for a microbatch. When this happens, I get a warning:
WARN MicroBatchExecution: Could not report metrics as number leaves in trigger logical plan did not match that of the execution plan:
logical plan leaves: (size = 8) OMITTED
execution plan leaves: (size = 7), FileScan csv OMITTED
Is there a way to avoid getting this warning, as this is an expected situation?