I'm using a Hadoop cluster with 3 nodes. When I try to write a file in my HDFS I'm getting this error:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /usr/local/hadoop/hdfs/data/datanode/javadeveloperzone/javareadwriteexample/read_write_hdfs_example.txt could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2219) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2789) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:892) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:574) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy14.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:510) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.addBlock(DFSOutputStream.java:1078) at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1865) at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1668) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
I checked the status of all my datanodes, but everything seems OK:
ubuntu@ip-172-31-3-210:~$ hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Configured Capacity: 24777043968 (23.08 GB)
Present Capacity: 9702780928 (9.04 GB)
DFS Remaining: 9702682624 (9.04 GB)
DFS Used: 98304 (96 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 172.31.3.210:9866 (ip-172-31-3-210.eu-west-3.compute.internal)
Hostname: ip-172-31-3-210.eu-west-3.compute.internal
Decommission Status : Normal
Configured Capacity: 8259014656 (7.69 GB)
DFS Used: 32768 (32 KB)
Non DFS Used: 5014663168 (4.67 GB)
DFS Remaining: 3227541504 (3.01 GB)
DFS Used%: 0.00%
DFS Remaining%: 39.08%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Jan 31 18:03:32 UTC 2021
Last Block Report: Sun Jan 31 17:51:56 UTC 2021
Num of Blocks: 0
Name: 172.31.3.238:9866 (ip-172-31-3-238.eu-west-3.compute.internal)
Hostname: ip-172-31-3-238.eu-west-3.compute.internal
Decommission Status : Normal
Configured Capacity: 8259014656 (7.69 GB)
DFS Used: 32768 (32 KB)
Non DFS Used: 5004750848 (4.66 GB)
DFS Remaining: 3237453824 (3.02 GB)
DFS Used%: 0.00%
DFS Remaining%: 39.20%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Jan 31 18:03:34 UTC 2021
Last Block Report: Sun Jan 31 17:51:52 UTC 2021
Num of Blocks: 0
Name: 172.31.4.242:9866 (ip-172-31-4-242.eu-west-3.compute.internal)
Hostname: ip-172-31-4-242.eu-west-3.compute.internal
Decommission Status : Normal
Configured Capacity: 8259014656 (7.69 GB)
DFS Used: 32768 (32 KB)
Non DFS Used: 5004517376 (4.66 GB)
DFS Remaining: 3237687296 (3.02 GB)
DFS Used%: 0.00%
DFS Remaining%: 39.20%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Jan 31 18:03:34 UTC 2021
Last Block Report: Sun Jan 31 17:51:52 UTC 2021
Num of Blocks: 0
Have you any idea to solve this issue please?
Thank you.
[1]: https://i.stack.imgur.com/B575q.png
[2]: https://i.stack.imgur.com/K1DgV.png
[3]: https://i.stack.imgur.com/NVLIx.png