Problem Could not find any valid local directory for s3ablock-0001-

user_6341672 0
Viewed 3988

I'm facing a problem running Jobs on an Amazon EMR when I try to write data on S3.

This is the stacktrace:

org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for s3ablock-0001-
    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:463)
    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:477)
    at org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:213)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:589)
    at org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
    at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
    at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.(S3ABlockOutputStream.java:168)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:822)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1105)
    at org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
    at org.apache.parquet.hadoop.ParquetFileWriter.(ParquetFileWriter.java:248)
    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:390)
    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:349)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.(ParquetOutputWriter.scala:37)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:158)
    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)
    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.(FileFormatDataWriter.scala:111)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:264)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:205)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I builded a image using Amazon ECS for EMR with Apache Spark 3.0.1 and Hadoop 3.2.1, I've already tried using Apache Spark 2.4.5 and Hadoop 2.7.1 with no success.

When we build up the EMR without ECS Image, manually, the job finished with success and writes everything it needs on S3.

I plead to you guys, what I need to do to put this things up and running? Thanks a lot.

amazon-emr pyspark apache-spark hadoop
3 Answers

you need to give the app a directory to store data in

spark.hadoop.fs.s3a.buffer.dir /tmp,/drive1/tmp

Normally it picks up what hadoop.tmp.dir is set to. Maybe you just don't have enough disk space, or the option is set to somewhere on a small root drive.

Better: include a entry for every disk you have, it will try to use any which has enough space.

Further reading How S3A writes data to S3

On Hadoop 3.2.1 you can tell S3A to buffer in heap or bytebuffer, so not use local disk at all.

spark.hadoop.fs.s3a.fast.upload.buffer bytebuffer

We do that in some deployments where the process doesn't have write access to the local FS and/or there's no capacity. But you then need to put in effort tuning some of the other related parameters to avoid buffering too much data -the limited bandwidth from EC2 to S3 can build up big backlogs

Actually, that may be the problem with disk buffering too -maybe you are just creating data faster than it can be uploaded. Try limiting the number of blocks which single output stream (here: spark worker thread) can have queued for upload before the stream writes block:

spark.hadoop.fs.s3a.fast.upload.active.blocks 1

That and/or smaller number of worker threads.

Please update this post which whatever worked, so others can make use your findings

user_22612740

Try to use offheap or bytebuffer instead of disk for S3A uploads. Most likely another process evicts s3ablock files during uploads.

I ran into this issue while writing to a delta table when delta io cache was enabled (read another delta table).

user_20882500

For me this was showing up due to a permission issue with the directory set in hadoop.tmp.dir. For example, when I changed the permissions to 777 for testing, my job started running successfully.

user_30616860
Related