Trying to execute a simple pyspark job on EMR (emr-6.2.0 and Spark 3.0.1) as follows:
spark = (SparkSession
.builder
.appName("Spark Downloader")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.dynamicAllocation.enabled", "true")
.config("spark.sql.streaming.schemaInference", "true")
.config("maximizeResourceAllocation", "true")
.getOrCreate())
spark.sparkContext.setLogLevel("WARN")
jdbcDf = (spark.read
.format("jdbc")
.option("url","jdbc:sqlserver://sqlserver:1433;databaseName=dw")
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
.option("dbtable", "db.dbo.My200GBTable")
.option("user", "xxx")
.option("password", "xxx")
.option("partitionColumn", "SOMEDATEFIELD")
.option("lowerbound", "2019-01-01")
.option("upperBound", "2022-05-26")
.option("numPartitions", 250).load())
jdbcDf.write.mode("overwrite").partitionBy("SOMEDATEFIELD").parquet("hdfs:///output/tables/My200GBTable/")
spark.stop()
Submitting with:
"Name": "1. Spark Download My200GBTable to HDFS",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
"--master",
"yarn",
"--conf",
"spark.jars.packages=com.microsoft.azure:spark-mssql-connector_2.12:1.2.0",
"--conf",
"spark.yarn.submit.waitAppCompletion=true",
"s3a://pyspark_apps/download_table_to_hdfs.py"
In another step I will use s3-dist-cp to send it to an S3 befor the cluster stops.
It works properly, connects and generate all the output down to the _SUCCESS file. However there is always a remaining task preventing the job to finish. Yarn timeline looks like:
The partitions are not skewed, generating evenly sizied files. Am I missing something obvious?
