I've recently started working in a code base with some legacy spark jobs. The general workflow looks something like this
val intermediate = df.sql('select ... from billions b join millions m on b.id1 = m.id1 or b.id2 = m.id2')
intermediate.persist()
intermediate.count()
val final = intermediate.sql('select ... from intermediate i join dim d on i.id = d.id')
final.persist()
final.count()
My initial impression was that the persist/count was a code smell, as it forced the realization of the dataset. However, when I remove them (individually or as a pair), I start seeing OOM errors and the job fails to complete.
I know the OR in the join condition could be problematic, as the data will likely need to be reshuffled.
More broadly though, in Spark jobs, is it common to force the realization of a dataset? In what cases might I want to do that? Are there common problems (in data or the spark code) that a novice spark developer might mask with data realization?