Spark Structured Streaming - Update following groupByKey and mapGroupsWithState giving duplicate key results

Viewed 70

I am trying to execute the following stateful aggregation in Databricks (scala):

sig_df
  .as[InputRow]
  .groupByKey(_.uid)
  .mapGroupsWithState(GroupStateTimeout.NoTimeout)(updateAcrossEvents)
  .writeStream
  .queryName("events_per_window_2")
  .format("memory")
  .outputMode("update")
  .start()

The functions managing the state are these:

    def updateAcrossEvents(uid: String,
                       inputs: Iterator[InputRow],
                       oldState: GroupState[UState]):UState = 
{
     var state:UState = if (oldState.exists) oldState.get else UState(uid, -999999, -999999, -999999)
 
  for (input <- inputs) {
    state = updateUStateWithEvent(state, input)
    oldState.update(state)
  }
  state
}

And this:

def updateUStateWithEvent(state:UState, input:InputRow):UState = {
// no timestamp, just ignore it  
if (Option(input.timestamp).isEmpty) {
    return state
  }  

if (input.sig_id == 10) {
  state.front_in = input.sig_value.toInt
}
  else if (input.sig_id == 17) {
   state.rear_in = input.sig_value.toInt
  }
  else if (input.sig_id == 25){
    state.top_in = input.sig_value.toInt
  }
  
  //return the updated state
  state
}

The issue I am facing is that the output has duplicates for the key uid. The following query returns plenty of results:

SELECT uid, count(*) FROM events_per_window_2
where front_in <> -999999
or rear_in <> -999999
or top_in <> -999999
group by uid
having count(*) > 1

I was of the understanding that since the outputMode is an update, we will not get any dupes.What might be going wrong with my approach here?

0 Answers
Related