AWS how to free up /dev/nvme1n1p1 and /dev/loop# deal with "No space left on device"

user_3608469 0
Viewed 1495

AWS newbie question. I created and run the AWS instance base on Ubuntu Server 18.04 LTS (HVM), SSD Volume Typefor bioinformatics. A size of the root volume was initially set at 8G and after installing dozens of packages, I get "No space left on device" error every time I update or upgrade packages. Then, I created AMI from this instance and increased the size of the root volume to 150G and launched this AMI but still get the same error. My filesystem size looks like following;

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            124G     0  124G   0% /dev
tmpfs            25G  956K   25G   1% /run
/dev/nvme1n1p1  7.7G  7.7G     0 100% /
tmpfs           124G     0  124G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           124G     0  124G   0% /sys/fs/cgroup
/dev/loop1       18M   18M     0 100% /snap/amazon-ssm-agent/1480
/dev/loop2       90M   90M     0 100% /snap/core/8268
/dev/loop0       94M   94M     0 100% /snap/core/8935
/dev/loop3       18M   18M     0 100% /snap/amazon-ssm-agent/1566
tmpfs            25G     0   25G   0% /run/user/1000

I guess the issue is the size of /dev/nvme1n1p1 and /dev/loop1..4, both which apparently are still 100% after increasing the size of the root volume. I have the option of start the whole process over again but I'd appreciate any pointers to get around this issue.

out-of-memory amazon-web-services amazon-ec2
1 Answers

I would suggest to first look over your data pipeline to see which processes are generating data, where it is stored and if you can optimize the data location and amount that is stored.

Second, I would use the results from the previous task to establish a data retention policy: Which data has to be stored and for how long? How often can you clean up data?

Then it is simple matter of creating a cron job to handle the data cleanup. Of course you also have to ask yourself if you want to have a manual check before data is wiped from the hard drive. Such a manual check can be created by triggering a sent email with cron and mail to a user of choice.

user_38577690
Related