In my EMR spark cluster in the core machines I have an EBS volume size of 128GB. My Spark program is giving me "No space left on device" error. So, what I want is in EMR to point the temp directory that spark uses to a different file system "/dev/xvdb2" . How can I do that ?
Below is the output from one of my core machine.
Filesystem Size Used Avail Use% Mounted on
devtmpfs 34G 78k 34G 1% /dev
tmpfs 34G 0 34G 0% /dev/shm
/dev/xvda1 11G 4.3G 6.1G 42% /
/dev/xvdb1 5.4G 35M 5.4G 1% /emr
/dev/xvdb2 133G 12G 121G 9% /mnt