Can some explain what is sc._jvm and sc._jsc in spark? For what purpose these 2 are used in spark?

user_17513842 0
Viewed 548

I work in spark with Python, What does the below code do?

jvm = sc._jvm
jsc = sc._jsc
fs = jvm.org.apache.hadoop.fs.FileSystem.get(jsc.hadoopConfiguration())
if fs.exists(jvm.org.apache.hadoop.fs.Path(loc+"_SUCCESS")):
    df=spark.read.parquet(loc)
    df.write.mode("overwrite").parquet(hdfs_bkp_loc)
else:
bigdata apache-spark jvm java
1 Answers

Since Spark is written in Scala, which is a JVM based language. PySpark has a JVM running behind the scenes to actually run the spark code. sc._jvm is the gateway into said JVM and sc._jsc is the Java Spark Context which is a proxy into the SparkContext in that JVM.

The snippet you posted basically calls some Java functions to check is some path exists and then read some data from there to be written into another location.

Using these _jsc members is discouraged and quite rare that you would need them.

user_28590540
Related