I'm trying to configure a 2 node cluster, where the Master Node is also a Data Node; and a Worker Data Node. Each node is a physical Windows machines running WSL-v2 Ubuntu 20.4 each.
- Master Node - 192.168.1.65 - DataScience
- Worker Node - 192.168.1.70 - NAS
I can get the Master Data Node to show in the web UI of Hadoop as a Data Node: Hadoop UI
But the Worker Data Node seems like it's unable to connect to the Master Node. In the log of the Worker Node I get this: DataNode WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
SSH is working, because the DataNode process gets started in the Worker Node, by executing start-dfs.sh on the MasterNode.
STARTUP_MSG: java = 1.8.0_342
************************************************************/
2022-09-13 11:16:15,300 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2022-09-13 11:16:15,556 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/home/ml/dfs/datanode
2022-09-13 11:16:15,633 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2022-09-13 11:16:15,697 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-09-13 11:16:15,697 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2022-09-13 11:16:15,850 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-09-13 11:16:15,862 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576
2022-09-13 11:16:15,867 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is NAS
2022-09-13 11:16:15,867 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-09-13 11:16:15,871 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0
2022-09-13 11:16:15,886 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:9866
2022-09-13 11:16:15,887 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth is 104857600 bytes/s
2022-09-13 11:16:15,887 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 100
2022-09-13 11:16:15,919 INFO org.eclipse.jetty.util.log: Logging initialized @1053ms to org.eclipse.jetty.util.log.Slf4jLog
2022-09-13 11:16:16,000 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/ml/hadoop-http-auth-signature-secret
2022-09-13 11:16:16,006 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined
2022-09-13 11:16:16,012 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2022-09-13 11:16:16,014 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2022-09-13 11:16:16,014 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2022-09-13 11:16:16,014 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2022-09-13 11:16:16,067 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 45959
2022-09-13 11:16:16,068 INFO org.eclipse.jetty.server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
2022-09-13 11:16:16,086 INFO org.eclipse.jetty.server.session: DefaultSessionIdManager workerName=node0
2022-09-13 11:16:16,086 INFO org.eclipse.jetty.server.session: No SessionScavenger set, using defaults2022-09-13 11:16:16,087 INFO org.eclipse.jetty.server.session: node0 Scavenging every 600000ms
2022-09-13 11:16:16,098 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2049a9c1{logs,/logs,file:///home/ml/hadoop/logs/,AVAILABLE}
2022-09-13 11:16:16,098 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7c51f34b{static,/static,file:///home/ml/hadoop/share/hadoop/hdfs/webapps/static/,AVAILABLE}
2022-09-13 11:16:16,153 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@53f6fd09{datanode,/,file:///home/ml/hadoop/share/hadoop/hdfs/webapps/datanode/,AVAILABLE}{file:/home/ml/hadoop/share/hadoop/hdfs/webapps/datanode}
2022-09-13 11:16:16,161 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@3e74829{HTTP/1.1, (http/1.1)}{localhost:45959}
2022-09-13 11:16:16,161 INFO org.eclipse.jetty.server.Server: Started @1296ms
2022-09-13 11:16:16,222 WARN org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Got null for restCsrfPreventionFilter - will not do any filtering.
2022-09-13 11:16:16,284 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:9864
2022-09-13 11:16:16,288 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
2022-09-13 11:16:16,289 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = ml
2022-09-13 11:16:16,289 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup
2022-09-13 11:16:16,335 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
2022-09-13 11:16:16,350 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9867
2022-09-13 11:16:16,497 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:9867
2022-09-13 11:16:16,523 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2022-09-13 11:16:16,529 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2022-09-13 11:16:16,537 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to DataScience/192.168.1.65:9000 starting to offer service
2022-09-13 11:16:16,542 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2022-09-13 11:16:16,542 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9867: starting
2022-09-13 11:16:18,616 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:16:25,660 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:16:32,722 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:16:39,777 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:16:46,824 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:16:53,866 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:00,913 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:07,959 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:14,995 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:22,038 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:29,095 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:36,146 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:43,207 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:50,391 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:17:57,444 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:04,471 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:11,506 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:18,553 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:25,597 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:32,657 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:39,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:46,793 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:18:53,873 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
2022-09-13 11:19:00,904 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: DataScience/192.168.1.65:9000
The LOG from the Master DataNode:
STARTUP_MSG: java = 1.8.0_342
************************************************************/
2022-09-13 11:16:11,067 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2022-09-13 11:16:11,377 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/home/ml/dfs/datanode
2022-09-13 11:16:11,453 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2022-09-13 11:16:11,511 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-09-13 11:16:11,511 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2022-09-13 11:16:11,657 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-09-13 11:16:11,671 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576
2022-09-13 11:16:11,675 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is DataScience
2022-09-13 11:16:11,675 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-09-13 11:16:11,679 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0
2022-09-13 11:16:11,694 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:9866
2022-09-13 11:16:11,695 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth is 104857600 bytes/s
2022-09-13 11:16:11,695 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 100
2022-09-13 11:16:11,727 INFO org.eclipse.jetty.util.log: Logging initialized @1153ms to org.eclipse.jetty.util.log.Slf4jLog
2022-09-13 11:16:11,810 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/ml/hadoop-http-auth-signature-secret
2022-09-13 11:16:11,816 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined
2022-09-13 11:16:11,823 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2022-09-13 11:16:11,824 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2022-09-13 11:16:11,824 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2022-09-13 11:16:11,824 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2022-09-13 11:16:11,890 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 42419
2022-09-13 11:16:11,891 INFO org.eclipse.jetty.server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
2022-09-13 11:16:11,909 INFO org.eclipse.jetty.server.session: DefaultSessionIdManager workerName=node0
2022-09-13 11:16:11,909 INFO org.eclipse.jetty.server.session: No SessionScavenger set, using defaults2022-09-13 11:16:11,910 INFO org.eclipse.jetty.server.session: node0 Scavenging every 600000ms
2022-09-13 11:16:11,920 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2049a9c1{logs,/logs,file:///home/ml/hadoop/logs/,AVAILABLE}
2022-09-13 11:16:11,921 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7c51f34b{static,/static,file:///home/ml/hadoop/share/hadoop/hdfs/webapps/static/,AVAILABLE}
2022-09-13 11:16:11,976 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@53f6fd09{datanode,/,file:///home/ml/hadoop/share/hadoop/hdfs/webapps/datanode/,AVAILABLE}{file:/home/ml/hadoop/share/hadoop/hdfs/webapps/datanode}
2022-09-13 11:16:11,982 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@3e74829{HTTP/1.1, (http/1.1)}{localhost:42419}
2022-09-13 11:16:11,982 INFO org.eclipse.jetty.server.Server: Started @1408ms
2022-09-13 11:16:12,056 WARN org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Got null for restCsrfPreventionFilter - will not do any filtering.
2022-09-13 11:16:12,128 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:9864
2022-09-13 11:16:12,132 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
2022-09-13 11:16:12,133 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = ml
2022-09-13 11:16:12,133 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup
2022-09-13 11:16:12,182 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
2022-09-13 11:16:12,197 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9867
2022-09-13 11:16:12,362 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:9867
2022-09-13 11:16:12,384 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2022-09-13 11:16:12,393 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2022-09-13 11:16:12,402 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to DataScience/127.0.1.1:9000 starting to offer service
2022-09-13 11:16:12,407 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2022-09-13 11:16:12,407 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9867: starting
2022-09-13 11:16:12,677 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to DataScience/127.0.1.1:9000
2022-09-13 11:16:12,679 INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
2022-09-13 11:16:12,689 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/ml/dfs/datanode/in_use.lock acquired by nodename 12583@DataScience
2022-09-13 11:16:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Analyzing storage directories for bpid BP-1514539440-127.0.1.1-1662735586830
2022-09-13 11:16:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled for /home/ml/dfs/datanode/current/BP-1514539440-127.0.1.1-1662735586830
2022-09-13 11:16:12,724 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=167770591;bpid=BP-1514539440-127.0.1.1-1662735586830;lv=-57;nsInfo=lv=-66;cid=CID-2488a887-bdf1-4c9e-8b5b-3792e28228b3;nsid=167770591;c=1662735586830;bpid=BP-1514539440-127.0.1.1-1662735586830;dnuuid=cebcadab-ea70-4a99-b46c-62ea9082769e
2022-09-13 11:16:12,740 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: The datanode lock is a read write lock
2022-09-13 11:16:12,828 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-31f50cfa-a0b1-4cc6-b667-5bbdaf796ac2
2022-09-13 11:16:12,828 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - [DISK]file:/home/ml/dfs/datanode, StorageType: DISK
2022-09-13 11:16:12,834 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader: Initializing cache loader: MemoryMappableBlockLoader.
2022-09-13 11:16:12,836 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean
2022-09-13 11:16:12,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1514539440-127.0.1.1-1662735586830
2022-09-13 11:16:12,843 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1514539440-127.0.1.1-1662735586830 on volume /home/ml/dfs/datanode...
2022-09-13 11:16:12,853 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Cached dfsUsed found for /home/ml/dfs/datanode/current/BP-1514539440-127.0.1.1-1662735586830/current: 32768
2022-09-13 11:16:12,870 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1514539440-127.0.1.1-1662735586830 on /home/ml/dfs/datanode: 26ms
2022-09-13 11:16:12,870 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1514539440-127.0.1.1-1662735586830: 28ms
2022-09-13 11:16:12,872 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-1514539440-127.0.1.1-1662735586830 on volume /home/ml/dfs/datanode...
2022-09-13 11:16:12,872 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice: Replica Cache file: /home/ml/dfs/datanode/current/BP-1514539440-127.0.1.1-1662735586830/current/replicas doesn't exist
2022-09-13 11:16:12,874 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1514539440-127.0.1.1-1662735586830 on volume /home/ml/dfs/datanode: 2ms
2022-09-13 11:16:12,874 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map for block pool BP-1514539440-127.0.1.1-1662735586830: 3ms
2022-09-13 11:16:12,875 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /home/ml/dfs/datanode
2022-09-13 11:16:12,885 INFO org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker: Scheduled health check for volume /home/ml/dfs/datanode
2022-09-13 11:16:12,899 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/home/ml/dfs/datanode, DS-31f50cfa-a0b1-4cc6-b667-5bbdaf796ac2): no suitable block pools found to scan. Waiting 1457882218 ms.
2022-09-13 11:16:12,903 WARN org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value above 1000 ms/sec. Assuming default value of -1
2022-09-13 11:16:12,904 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting in 13165142ms with interval of 21600000ms and throttle limit of -1ms/s
2022-09-13 11:16:12,911 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1514539440-127.0.1.1-1662735586830 (Datanode Uuid cebcadab-ea70-4a99-b46c-62ea9082769e) service to DataScience/127.0.1.1:9000 beginning handshake with NN
2022-09-13 11:16:13,040 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1514539440-127.0.1.1-1662735586830 (Datanode Uuid cebcadab-ea70-4a99-b46c-62ea9082769e) service to DataScience/127.0.1.1:9000 successfully registered with NN
2022-09-13 11:16:13,040 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode DataScience/127.0.1.1:9000 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000
2022-09-13 11:16:13,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0x6781f639d6e11e3b with lease ID 0x8d7a159c7ac2b265 to namenode: DataScience/127.0.1.1:9000, containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 3 msecs to generate and 48 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
2022-09-13 11:16:13,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Got finalize command for block pool BP-1514539440-127.0.1.1-1662735586830
I did port forwarded this ports in each machine, and opened the ports in Windows firewall:
Master Node port forwarded ports
Worker Node port forwarded ports
If I do a nc -vz for each of them I get this: Open ports in each machine
This is the core-site.xml:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://DataScience:9000</value>
</property>
</configuration>
This is the hdfs-site.xml:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/ml/dfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/ml/dfs/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
Master Node /etc/hosts:
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1 localhost
127.0.1.1 DataScience.localdomain DataScience
# I ADDED THIS
192.168.1.65 DataScience
192.168.1.70 NAS
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Worker Node /etc/hosts:
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1 localhost
127.0.1.1 NAS.localdomain NAS
# I ADDED THIS
192.168.1.65 DataScience
192.168.1.70 NAS
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
What i'm missing, thanks in advance.