looking local file system instead of hdfs when I ran the spark submit

user_5680504 0
Viewed 1643

when I ran the spark-submit, it throw error indicating that no file in the file system as below.

Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/user/sclee/clustering2/mapTemplate_micron

I think that my file is on the hdfs not in the my local system. I found that my hadoop configuration file was correctly configured as below

<property>
<name>fs.defaultFS</name>
<value>hdfs://spark.dso.hdm1:9000</value>
</property>
<property>

How to resolve this issue?

supplement

Below is my submit query. Actually, I used the spark well using below query. However, I mistakenly removed the spark directories. So I copied the spark directory from worker node. And then my issue occurred. I hope to fix my issue. Thanks.

hadoop fs -rm -r /home/hawq2/*
spark-submit \
        --class com.bistel.spark.examples.yma.ClusterServiceBasedOnNewAlgo \
        --master spark://spark.dso.spkm1:7077 \
        --executor-memory 8g\
        --executor-cores 4\
        --jars /home/jumbo/user/sclee/clustering/guava-19.0.jar\
        --conf spark.eventLog.enabled=true\
        --conf spark.eventLog.dir=hdfs://spark.dso.hdm1:9000/user/jumbo/applicationHistory\
        --conf spark.memory.offHeap.enabled=true\
        --conf spark.memory.offHeap.size=268435456\
        ./new.jar\
        /user/sclee/clustering2/mapTemplate_micron      /user/sclee/clustering2/data/bin3       /user/sclee/clustering2/ret
apache-spark
1 Answers

It looks your HADOOP_CONF_DIR isn't loaded, or the files in it

For example, check this in the spark-env.sh, setting the correct directory for your config 

HADOOP_CONF_DIR=/etc/hadoop/

Then, ensure that you have configured hdfs-site.xml, core-site.xml, and yarn-site.xml in that directory. (Although looks like you're not using YARN, so probably just the core and hdfs)

user_23086830
Related