hadoop error: No MD5 file found corresponding to image file

user_1568658 0
Viewed 136

Got an old hadoop system (that haven't been used for years), when trying to restart the cluster (1 master, 2 slaves), all on Linux, got error, on the namenode.

Error output:

2021-03-18 20:18:28,628 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: Failed to load image from FSImageFile(file=/home/xxx/tmp/hadoop/name/current/fsimage_0000000000000480607, cpktTxId=0000000000000480607)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:651)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:264)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
Caused by: java.io.IOException: No MD5 file found corresponding to image file /home/xxx/tmp/hadoop/name/current/fsimage_0000000000000480607
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:736)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:632)
        ... 9 more
2021-03-18 20:18:28,631 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2021-03-18 20:18:28,633 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

More info:

  • One of the slave's datanode's partition has bad disk blocks, so I have removed that partition from /etc/fstab so that to bring the Linux up. So, that slave's data is lost.

What I have tried:

  • Start the cluster, including the all 3 nodes, got above error.
  • Start the cluster, excluding the bad slave, thus only 2 nodes, still got above error.

Questions:

  • A. What the error means ?
  • B. Is it relevant to the bad slave?
  • C. Is there anyway to recover without re-format hdfs filesystem on namenode?
hdfs hadoop java
1 Answers

There should be a file called:

/home/xxx/tmp/hadoop/name/current/fsimage_0000000000000480607.md5

In the same location as the image file. It will have contents that look like this:

177e5f4ed0b7f43eb9e274903e069da4 *fsimage_0000000000000014367

Simply get the md5 sum of your fsimage file:

md5sum fsimage_0000000000000480607.md5

Then create a new md5 file that looks like:

xxxxxx *fsimage_0000000000000480607.md5

Where xxxxxx is the md5sum from the md5 command.

user_888390
Related