Run PySpark job from .egg instead of .py

user_3983547 0
Viewed 919

I am trying to run a PySpark Job using Dataproc. The only different thing comparing to all the examples out there is that I want to submit the job from .egg instead of .py file.

In order to submit the PySpark job in a regular commodity cluster would be something like: 

spark2-submit --master yarn \
    --driver-memory 20g \
    --deploy-mode client \
    --conf parquet.compression=SNAPPY \
    --jars spark-avro_2.11-3.2.0.jar \
    --py-files dummyproject-1_spark-py2.7.egg \
    dummyproject-1_spark-py2.7.egg#__main__.py "param1" "param2"

Now, I want to submit exactly the same job but using Dataproc. In order to accomplish this I am using the following command: 

gcloud dataproc jobs submit pyspark \
    file:///dummyproject-1_spark-py2.7.egg#__main__.py \
    --cluster=my-cluster-001 \
    --py-files=file:///dummyproject-1_spark-py2.7.egg

The error I am getting is:

Error: Cannot load main class from JAR file:/dummyproject-1_spark-py2.7.egg

It is important to mention that when I try to run a simple PySpark job using .py file, it is working correctly.

Can somebody tell me, how can I run a PySpark job from .egg file instead of .py file?

google-cloud-dataproc pyspark google-cloud-platform
1 Answers

It looks like there is a bug on how gcloud dataproc is parsing the arguments and making Spark try to execute your file like a Java JAR file. A workaround is copy your __main__.py file outside of your egg file and execute it independently like this. 

gcloud dataproc jobs submit pyspark \
    --cluster=my-cluster-001 \
    --py-files=file:///dummyproject-1_spark-py2.7.egg \
    file:///__main__.py \
user_38294260
Related