How to figure out hdfs URI - java.io.IOException: Incomplete HDFS URI, no host

user_12208910 0
Viewed 677

How can I figure out the URI my hdfs dfs commands are connecting to?

Is there any configuration file that stores the URI or any command that can be used to display it?

I looked into the documention of FileSystemShell and the dfsadmin documentation without success. (Also, I do not have access to most of dfsadmin commands.)

When I call a command with hdfs:///user/myUserName/... it throws the exception:

Exception in thread "main" java.io.IOException: Incomplete HDFS URI, no host: hdfs:///user/myUserName/test.avro
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.avro.mapred.FsInput.<init>(FsInput.java:38)
        at org.apache.avro.tool.Util.openSeekableFromFS(Util.java:110)
        at org.apache.avro.tool.DataFileGetSchemaTool.run(DataFileGetSchemaTool.java:47)
        at org.apache.avro.tool.Main.run(Main.java:87)
        at org.apache.avro.tool.Main.main(Main.java:76)

Simple commands like hdfs dfs -ls are working fine.

Using Hadoop 3.1.0.

hdfs hadoop
1 Answers

If you're able to access the file core-site.xml, then you can look for the value assigned to property fs.defaultFS

$ grep -A 2 defaultFS /etc/hadoop/conf/core-site.xml
    <name>fs.defaultFS</name>
    <value>hdfs://bigdataserver-2.internal.cloudapp.net:8020</value>
  </property>

Note: I use Cloudera and core-site.xml is where i get the detail. For Hadoop, you might be having core-default.xml

Check this out: https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-common/core-default.xml

user_3610890
Related