HDFS behavior when removing a DataNode directory

user_4846215 0
Viewed 657

I setup a Hadoop cluster (HDP 2.6) which contains of 3 nodes with the following HDFS mounting points on each node.

  • /mnt/datadisk1
  • /mnt/datadisk2
  • /mnt/datadisk3

So, my /etc/fstab file looks like this on each of the three nodes:

/dev/mapper/centos-home /home                   xfs     defaults        0 0
...
# Here the HDFS Partitions:
/dev/sdb                /mnt/datadisk1          xfs     defaults        0 0
/dev/sdc                /mnt/datadisk2          xfs     defaults        0 0
/dev/sdd                /mnt/datadisk3          xfs     defaults        0 0

The paths are also set in the HDFS config (Ambari) dfs.datanode.data.dir = /mnt/datadisk1,/mnt/datadisk2,/mnt/datadisk3.

My question is now: What would happen, if I would remove e.g. /mnt/datadisk3 from one of my Hadoop nodes? Would the HDFS be rebalanced and the data that was on this partition re-replicated? Would the /mnt/datadisk3 partition on the other nodes still be usable?

Each partition has a size of ~1 TB. Would my HDFS afterwards have 3*1TB + 3*1TB + 2*1TB = 8TB or maybe only 2*1TB + 2*1TB + 2*1TB = 6TB (because all datadisk3 partitions would be removed from the HDFS)?

hdfs hadoop
2 Answers

  • If you remove or won't mention the path of 

    dfs.datanode.data.dir =/mnt/datadisk1,/mnt/datadisk2,/mnt/datadisk3

dfs.datanode.data.dir = /mnt/datadisk1,/mnt/datadisk2

    their won't be any connection with datadisk3,so datadisk3 data no more available.

  • Yes , HDFS be rebalance the data that was on respective partition and replicate it into other block(based on your replication factor).

  • no you can't use data that is stored in datadisk3 (if you are using commissioning and decommissioning method then only you can reuse the data)

user_75271890

If you have 3 nodes and you have deleted one. this means there are 2 nodes left and each node contains same data since replication factor is 2, but your data will be there but with only two replicas

user_93680700
Related