I am trying to execute the following stateful aggregation in Databricks (scala):
sig_df
.as[InputRow]
.groupByKey(_.uid)
.mapGroupsWithState(GroupStateTimeout.NoTimeout)(updateAcrossEvents)
.writeStream
.queryName("events_per_window_2")
.format("memory")
.outputMode("update")
.start()
The functions managing the state are these:
def updateAcrossEvents(uid: String,
inputs: Iterator[InputRow],
oldState: GroupState[UState]):UState =
{
var state:UState = if (oldState.exists) oldState.get else UState(uid, -999999, -999999, -999999)
for (input <- inputs) {
state = updateUStateWithEvent(state, input)
oldState.update(state)
}
state
}
And this:
def updateUStateWithEvent(state:UState, input:InputRow):UState = {
// no timestamp, just ignore it
if (Option(input.timestamp).isEmpty) {
return state
}
if (input.sig_id == 10) {
state.front_in = input.sig_value.toInt
}
else if (input.sig_id == 17) {
state.rear_in = input.sig_value.toInt
}
else if (input.sig_id == 25){
state.top_in = input.sig_value.toInt
}
//return the updated state
state
}
The issue I am facing is that the output has duplicates for the key uid. The following query returns plenty of results:
SELECT uid, count(*) FROM events_per_window_2
where front_in <> -999999
or rear_in <> -999999
or top_in <> -999999
group by uid
having count(*) > 1
I was of the understanding that since the outputMode is an update, we will not get any dupes.What might be going wrong with my approach here?