You can set the size of the Eden to be an over-estimate of how much memory each task will need. If the size of Eden is determined to be E, you can set the size of the Young generation using the option -Xmn=4/3*E. (The scaling up by 4/3 is to account for space used by survivor regions, as well.) As an example, if your task is reading data from HDFS, the amount of memory used by the task can be estimated by using the size of the data block read from HDFS. Note that the size of a decompressed block is often two or three times the size of the block. "
(from "Spark: The Definitive Guide: Big Data Processing Made Simple (English Edition) by Bill Chambers, Matei Zaharia)
Can i assume that, even if I have a big file (csv, avro, parquet or orc) the maximum amount of data of a partition that will be put in memory is the size of block read?
Or the maximum amount of data of a partition that will be put in memory is the value of spark.files.maxPartitionBytes which by default is 128 MB?
What's happen if I add a map operation next to read operantion, do i need to estimate how much memory a map operation (e.g. splitting a string into key-value par like (work,count)) will use or the over-estimated space for survivor regions will be enough?
Seems that spark.files.maxPartitionBytes act as setting of safety that limits that any read operation (the initial or any partitions resulting from any transformation) so that we know that any transformation and any task will read and put into memory the maximum of memory specified in this setting. Am I right?