I am slightly confused as to how does spark reads the data from s3 for example. Let's say there is 100 GB of data to be read from s3 and the spark cluster has a total memory of 30 GB. Will spark read all 100 GB of the data once an action is triggered and store the maximum number of partitions in memory and spill the rest to disk or will it read only the partitions that it can store in memory process them and then read the rest of the data? Any link to some documentation will be highly appreciated.
1 Answers
There is a question on Spark FAQ about this:
Does my data need to fit in memory to use Spark?
No. Spark's operators spill data to disk if it does not fit in memory, allowing it to run well on any sized data. Likewise, cached datasets that do not fit in memory are either spilled to disk or recomputed on the fly when needed, as determined by the RDD's storage level.
MEMORY_AND_DISK
Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed.
Related