Pyspark on EMR with JDBC task hangs forever

Viewed 71

Trying to execute a simple pyspark job on EMR (emr-6.2.0 and Spark 3.0.1) as follows:

spark = (SparkSession 
        .builder
        .appName("Spark Downloader")
        .config("spark.sql.adaptive.enabled", "true")
        .config("spark.dynamicAllocation.enabled", "true")
        .config("spark.sql.streaming.schemaInference", "true")
        .config("maximizeResourceAllocation", "true")
        .getOrCreate())

spark.sparkContext.setLogLevel("WARN")

jdbcDf = (spark.read
         .format("jdbc")
         .option("url","jdbc:sqlserver://sqlserver:1433;databaseName=dw")
         .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
         .option("dbtable", "db.dbo.My200GBTable")
         .option("user", "xxx")
         .option("password", "xxx")
         .option("partitionColumn", "SOMEDATEFIELD")
         .option("lowerbound", "2019-01-01")
         .option("upperBound", "2022-05-26")
         .option("numPartitions", 250).load())

jdbcDf.write.mode("overwrite").partitionBy("SOMEDATEFIELD").parquet("hdfs:///output/tables/My200GBTable/")

spark.stop()

Submitting with:

"Name": "1. Spark Download My200GBTable to HDFS",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
    "Jar": "command-runner.jar",
    "Args": [
        "spark-submit",
        "--deploy-mode",
        "cluster",
        "--master",
        "yarn",
        "--conf",
        "spark.jars.packages=com.microsoft.azure:spark-mssql-connector_2.12:1.2.0",
        "--conf",
        "spark.yarn.submit.waitAppCompletion=true",
        "s3a://pyspark_apps/download_table_to_hdfs.py"

In another step I will use s3-dist-cp to send it to an S3 befor the cluster stops.

It works properly, connects and generate all the output down to the _SUCCESS file. However there is always a remaining task preventing the job to finish. Yarn timeline looks like:

yarn timeline

The partitions are not skewed, generating evenly sizied files. Am I missing something obvious?

0 Answers
Related