I'm trying to run the following command to submit a pyspark job on EMR with my own libraries and public libraries (such as pandas and numpy):
spark-submit --deploy-mode client --py-files dependencies.zip main.py
The dependencies.zip has pandas library and its dependencies, and my internal library (xpto library):
- dependencies:
- pandas
- numpy
- ...
- xpto
- __init__.py
- xpto.py
When I start the job I receive the error: no module named 'pandas'.
So, How Can I pass my own libraries and public libraries within a .zip to run the job?