Spark Structured Streaming warning when applying union to datasets

Viewed 306

In my Spark Structured Streaming application, I'm doing unionByName on three Datasets:

val allRecords = ds1
  .unionByName(ds2)
  .unionByName(ds3)

where ds1 etc are strongly-typed Datasets. Sometimes, one Dataset is empty for a microbatch. When this happens, I get a warning:

WARN MicroBatchExecution: Could not report metrics as number leaves in trigger logical plan did not match that of the execution plan:
logical plan leaves: (size = 8) OMITTED
execution plan leaves: (size = 7), FileScan csv OMITTED

Is there a way to avoid getting this warning, as this is an expected situation?

0 Answers
Related