pyspark - spark-submit with internal and public libraries

Viewed 104

I'm trying to run the following command to submit a pyspark job on EMR with my own libraries and public libraries (such as pandas and numpy):

spark-submit --deploy-mode client --py-files dependencies.zip main.py

The dependencies.zip has pandas library and its dependencies, and my internal library (xpto library):

- dependencies:
  - pandas
  - numpy
  - ...
  - xpto
     - __init__.py
     - xpto.py

When I start the job I receive the error: no module named 'pandas'.

So, How Can I pass my own libraries and public libraries within a .zip to run the job?

0 Answers
Related