Pyspark and Cassandra secure bundle.zip

user_3421834 0
Viewed 124

I am new to this pyspark cassandra technology, I have found a lot of resources about Spark.sql in pyspark and I think that's cool, but I am still stuck at the level of connection with my database because I need a secure-bundle.zip from datastax to connect. So, here I go with my questions:

I understand this is the way to connect to my remote database with pyspark shell:

pyspark --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1\ 
        --files path_to/secure-connect-test.zip \
        --conf spark.cassandra.connection.config.cloud.path=secure-connect-test.zip \
        --conf spark.cassandra.auth.username=UserName \
        --conf spark.cassandra.auth.password=Password \
        --conf spark.dse.continuousPagingEnabled=false

I have downloaded the jar files for com.datastax.spark:spark-cassandra-connector_2.11:2.5.1, so I have copied those jars to my spark/jars folder (that's correct, right?)

Then , my spark shell commands end like:

pyspark  --files path_to/secure-connect-test.zip \
        --conf spark.cassandra.connection.config.cloud.path=secure-connect-test.zip \
        --conf spark.cassandra.auth.username=UserName \
        --conf spark.cassandra.auth.password=Password \
        --conf spark.dse.continuousPagingEnabled=false

Now, I understand all pyspark code is an API to controll spark, then, all commands above can be rewritten with pyspark code, the --conf parameters I understand are related to .config(value, value):

sparkSession=SparkSession.builder.appName('SparkCassandraApp').config('spark.cassandra.connection.host', 'localhost')

So, my question is, how to add my zip file?

I have seen the the way to upload a file to my spark session, in pyspark code, would be with SparkContext.addFile() but I am confused because I have to set the SparkContext before the Spark Session, so do I have to set the SparkContext , add my file and then add the SparkSession again even if I repeat same values? or can I add the bundle.zip file just in the Spark Session by adding the source path?

Please help, I just want to have a clear vision of spark and its way to work

Thanks

spark-cassandra-connector pyspark apache-spark cassandra python
1 Answers

if you're using --files, then Spark will upload the local file to the cluster and make it available to all executors, like it's done via .addFile. In this case you can just specify filename via spark.cassandra.connection.config.cloud.path. You don't need to copy file to jars folder, etc. - it could lay where it was downloaded.

as described in the blog post that I referenced, file could be also located on any location reachable from all executors, such as, S3/HDFS/HTTP/... - in this case, spark.cassandra.connection.config.cloud.path could just point to that location without need to specify via --files. If you copied file to all executors, then you can also use full path to that file as file://path-to-secure-bundle

Alex_Ott_186270
Related