Can we use flatMapGroupsWithState in batch mode?
We have been using flatMapGroupsWithState API to do sessionization of our click stream data. Now we are planning to use it to run in Batch Mode.
For more context check this video posted by Bartosz Konieczny in youtube Using Apache Spark to Solve Sessionization Problem in Batch and Streaming
Can we get rid off the full-outer join and use mapGroupWithState with a checkpoint folder and Spark State
Can we get rid off the full-outer join and use mapGroupWithState with a checkpoint folder and Spark State
Yes, if the source is supported in Structured Streaming. You can add the Trigger.Once and coordinate the execution from your data orchetsration layer in "batch" mode.
But since every job execution will start by loading all active state from the state store to Spark shuffle partitions, in the end it's quite similar to performing full-outer join with still active sessions. Unless, you want to reduce the latency and use streaming with a controlled execution? Can you explain a bit why you want to switch?
Best, Bartosz.
