I am ingesting a dataframe from a datasource (df_example), has two columns (col1: string, col2: string)
df_example.count()
Out[14]: 116020036
df_example.rdd.getNumPartions()
Out[15]: 20
spark.sparkContext.defaultParallelism
Out[16]: 8
spark.conf.get("spark.sql.shuffle.partitions")
Out[17]: '200'
I need to write this dataframe in Azure SQL DB I am executing the below code its taking very long
df_example.write \
.format("com.microsoft.sqlserver.jdbc.spark") \
.mode("overwrite") \
.option("url", url) \
.option("dbtable", "dbo.example") \
.option("accessToken", access_token) \
.option("encrypt", "true") \
.option("hostNameInCertificate", "*.database.windows.net") \
.option("tableLock", "true") \
.option("reliabilityLevel", "BEST_EFFORT") \
.save()
I see 20 tasks for this Job
4.3h have passed only 1/20 completed
Data written 32.8 MiB/2.3 GiB