Disk space issue in AWS EMR Cluster

user_4487066 0
Viewed 7972

We have provisioned 11 nodes(1 master + 10 cores) EMR cluster in AWS. We have chosen disk space for each node as 100 GB.

When the cluster is provisioned, the EMR automatically allocated only 10GB to root partition(/dev/xvda1). After some days root partition disk space becomes full, due to this we couldn't run any job or install basic softwares like git using yum command. 

[hadoop@<<ip address>> ~]$ df -BG
Filesystem     1G-blocks  Used Available Use% Mounted on
devtmpfs             79G    1G       79G   1% /dev
tmpfs                79G    0G       79G   0% /dev/shm
/dev/xvda1           10G    10G       0G  100% /
/dev/xvdb1            5G    1G        5G   4% /emr
/dev/xvdb2           95G   12G       84G  12% /mnt
/dev/xvdf            99G   12G       83G  12% /data

Could you please help us, how to resolve this issue?

  1. How to increase root partition(/dev/xvda1) disk space to 30GB?
  2. By default all installation using yum or rpm goes to root partition(/dev/xvda1). How to by-pass softwares installing to root partition(/dev/xvda1)?
  3. Whatever the solution, it should not disturb the existing EMR installation.

Help would be much appreciated.

emr amazon-emr yum amazon-web-services linux
4 Answers

Recently ran into same issue. Find the corresponding ec2 instance and in description tab find and click on the link root device. It points to a EBS Id, click on it. In the actions click on modify volume. After requesting required total space. you might have to aditionally run commands such as "growpart" to let the os adjust to the new size.

user_48511720
user_8332450
Related