I am using hadoop-2.6.0 on Ubuntu 16.04.3 LTS, and everything went well in my test env. But while running in the production env, my HDFS repeatedly complains about not having enough space for namenode, and HDFS has entered safemode.
2017-11-25 14:00:48,731 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on available disk space. Already in safe mode.
2017-11-25 14:00:48,731 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is ONResources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
2017-11-25 14:00:53,732 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume '/home/hadoop/.Private' is 0, which is below the configured reserved a mount 104857600
But the truth is I have more than enough space
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 33M 13G 1% /run
/dev/mapper/hdfsmaster--vg-root 7.1T 1.3T 5.6T 19% /
tmpfs 63G 4.0K 63G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sda2 473M 55M 394M 13% /boot
/dev/sda1 511M 4.0K 511M 1% /boot/efi
/home/hadoop/.Private 7.1T 1.3T 5.6T 19% /home/hadoop
HDFS data/name directories are all in /home/hadoop directory
HDFS can be started successfully, and it's NOT in safemode at beginning. But after some time (like 5, 6 hours), even if I did not perform any operation (read / write / query ), it will complain about lacking of space and enter safemode automatically.
I have tried:
- restart hdfs
- restart the server
- re-format namenode
- change data/name folder and restart hdfs
none of the above method helps.
Please help! This is killing me!