How to start a standalone cluster using pyspark?

user_4406595 0
Viewed 2499

I am using pyspark under ubuntu with python 2.7 I installed it using

pip install pyspark --user

And trying to follow the instruction to setup spark cluster

I can't find the script start-master.sh I assume that it has to do with the fact that i installed pyspark and not regular spark

I found here that i can connect a worker node to the master via pyspark, but how do i start the master node with pyspark?

pyspark apache-spark python
3 Answers

If you are already using pyspark through conda / pip installation, there's no need to install Spark and setup environment variables again for cluster setup.

For conda / pip pyspark installation is missing only 'conf', 'sbin' , 'kubernetes', 'yarn' folders, You can simply download Spark and move those folders into the folder where pyspark is located (usually site-packages folder inside python).

user_104843830
Related