when I try to read hive table, it gives GC overhead limit exceed error. I already tried to change the spark.executor.memory and spark.driver.memory but issue continuously coming up.
spark = SparkSession\
.builder\
.appName ("test")
.config("spark. executor.memory","20g"))\
.config("spark.network.timeout","200000s")\
.config("spark.master","yarn")\
.conFig("spark. submit.deployMode","client")\
.config("spark.driver.memory","25g")\
.config("spark. executor. instances","20")\
.config("spark. speculation","true")\
.config("spark.driver.maxResultsize","50g")\
.config("spark.sql.parquet.binaryAsstring","true")\
.config("spark.sql.broadcastrimeout","20000000ms")\
.config("spark.core.connection.ack.wait.timeout","6000s")\
.config("spark.driver.extraJavaOptions","-XX:MaxDirectMemorysize=999999998m-XX:+UseConcMarkSweepGC-XX:+CMSParallelRemarkEnabled XX:+UseCMSIniti")
.config("spark. executor.extraJavaOptions","-XX:MaxDirectMemorysize=999999998m -XX:+UseConcMarkSweepGC-XX:+CMSParallelRemarkEnabled-XX:+UseCMSIni")
.config ("spark.yarn.dist.files","/opt/mapr/hive/hive/conf/hive-site.xml")\
.config ("spark.sql.catalogImplementation","hive")\
.config ("spark.jars","/opt/pltf/cloak/lib/cloak-spark.jar,/app/spy/hive-contrib.jar")\
.config ("spark. yarn. archive","/opt/pltf/cloak/lib/spark-jars.zip")\
.config ("spark. shuffle.compress", "true")\
.config ("spark.shuffle.spill.compress", "true")\
.config ('spark. shuffle.io.maxRetries",1101) \
.config ("spark.sql.shuffle.partitions","1200" )\
.config ('spark.default.parallelism', '60000) \
.config ('spark.sql. shuffle.partitions' ,3000) \
. config ('spark.shuffle.registration.timeout', '10000 ) \
.config (spark. shuffle.registration.maxAttempts',5)\
.conFig ('spark.dynamicAllocation.minExecutors",'41)\
.config ('spark. yarn.queue', 'root-digviz')\
.config ('spark.memory. fraction','.7')\
.config ('spark.memory. storageFraction' ,'.5' )\
.enableHiveSupport()
getorCreate()
sc = spark.sparkContext
df = spark.sql("select * from db.emp_table where month=202206")
please help to set the correct configuration to resolve the GC issue.
Error Snapshot
