Hadoop errorcode -1000, No space available in any of the local directories

user_14639026 0
Viewed 921

I'm using Windows 7 with Hadoop 2.10.1 installed as shown here: https://exitcondition.com/install-hadoop-windows/ and I get an error when running my job:

INFO mapreduce.Job: 
Job job_1605374051781_0001 failed with state FAILED due to: 
Application application_1605374051781_0001 failed 2 times 
due to AM Container for appattempt_1605374051781_0001_000002 exited with 
exitCode: -1000 Failing this attempt.Diagnostics: 
[2020-11-14 18:17:54.217]No space available in any of the local directories.

The expected output is several lines of text and my disks are nowhere near full (at least 10GB free). The code is some generic mapreduce job that I cannot post here because it's the intellectual property of the university.

Any tips on how to solve the "No space available" error? For clarification I'm using only my PC, I'm not connected to other machines.

PS: I've solved it, as said here: Hadoop map reduce example stuck on Running job by user "banu reddy" https://stackoverflow.com/users/4249076/banu-reddy the free HDD space needs to be at least 10% od the disk.

hadoop
1 Answers

Hadoop's jobs are executed within the framework's distributed filesystem aka HDFS, which works independently from the local filesystem (even by operating in just one machine, as you clarified).

That basically means that the error you got referred to the disk space available in the HDFS and not on your hard drives in general. To check if the HDFS has enough disk space to run the job or not, you can execute the following command on the terminal:

hdfs dfs -df -h

Which can have an output like this (ignoring the warning I get on my Hadoop setup): enter image description here

If the command output in your system indicates that the available disk space is low or non-existent, you can individualy delete directories from the HDFS by firstly checking what directories and files are stored:

hadoop fs -ls

And then deleting each directory from the HDFS:

hadoop fs -rm -r name_of_the_folder

Or file from the HDFS:

hadoop fs -rm name_of_the_file

Alternatively, you can empty everything stored in the HDFS to be sure that you will not hit the disk space limit again any time soon. You can do that by stopping the YARN and HDFS daemons at first:

stop-all.sh

Then enabling only the HDFS daemon:

start-dfs.sh

Then formatting everything on the namenode (aka the HDFS in your system, not your local files of course):

hadoop namenode -format

And enabling YARN and HDFS daemons at last:

start-all.sh

Remember to re-run the hdfs dfs -df -h command after deleting stuff in the HDFS so you make sure you have free space on the HDFS.

user_56440370
Related