I'm using Spark Structured Streaming (PySpark) to join a Kafka Stream with a Static DataFrame using a MinHashLSH approxSimilarityJoin (which under the hood does a SortMergeJoin).
The size and volume of these messages is fairly high, which results in a state of multiple GBs after a couple hours. After a while the causes OOMerrors and crashes the process.
According to the docs, stream-static joins are stateless so I would expect no state to be accumulated over time. In this use case, we are also not interested in preserving any state other than the Kafka offset.
This image below shows the State size after a few minutes, the left side of the SortMergeJoin is the static frame, while the right side is the Kafka Stream. No watermarks or dropduplicates have been added along the way.
Does anyone have an idea on how I can reduce the size of this state or even get rid of it all together?
