Spark error when running TPCDS benchmark datasets - Could not find dsdgen

user_8362086 0
Viewed 600

Im trying to build the TPCDS benchmark datasets, by following this website.

https://xuechendi.github.io/2019/07/12/Prepare-TPCDS-For-Spark

when I run this:

scala> [troberts@master1 spark-sql-perf]$ spark-shell --master yarn --deploy-mode cliers /home/troberts/spark-sql-perf/target/scala-2.11/spark-sql-perf_2.11-0.5.1-SNAPSHOT.jar -i TPCDPreparation.scala

I get this error? Im wondering if its something to do with permissions as the file dsdgen definitely exists at that location on each of the worker nodes /home/troberts/spark-sql-perf/tpcds-kit/tools

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Aborting TaskSet 0.0 because task 0 (partition 0) cannot run anywhere due to node and executor blacklist. Most recent failure: Lost task 0.0 in stage 0.0 (TID 0, worker1.mycluster.com, executor 1): java.lang.RuntimeException: Could not find dsdgen at /home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen or //home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen. Run install at scala.sys.package$.error(package.scala:27)

Any ideas appreciated.

Cheers

apache-spark scala
1 Answers
Could not find dsdgen at /home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen or //home/troberts/spark-sql-perf/tpcds-kit/tools/dsdgen

You need to have TPCDS installed first.

spark-sql-perf docs from tool you've used:

Before running any query, a dataset needs to be setup by creating a Benchmark object. 
Generating the TPCDS data requires dsdgen built and available on the machines. 
We have a fork of dsdgen that you will need. 

The fork includes changes to generate TPCDS data to stdout, so that this library can pipe them directly to Spark, without intermediate files. 
Therefore, this library will not work with the vanilla TPCDS kit.

TPCDS kit needs to be installed on all cluster executor nodes under the same path!

Please, configure TPCDC toolkit from databricks

user_30691250
Related