For spark structured streaming we have written a custom source reader to read data from our "custom source". To write it , I was following the example of "kafka custom source" code in spark code itself. We also have a requirement of writing our output to a custom location. For that I saw that spark provides "foreach" and "foreachbatch"(in spark 2.3.0 onwards). I found that using either of these is a very simple implementation and also at first look , I feel most of my custom sink implementation requirement can be met.
But when I looked at kafka code in spark, I saw that kafka uses "StreamSinkProvider"(https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-StreamSinkProvider.html) instead of using "foreach" or "foreachbatch".
Now I wanted to find out what are various pros/cons of using wither of these options(in terms of performance/ flexibility etc.) ? Is one better than another ? Anyone have any experience working on either of these options how they go along in actual use case ?