I have inherited a web application that delegates some of its work to other machines via spark-submit.
spark-submit -v --py-files /somedir/python/pyspark_utils*.egg --master yarn /somedir/python/driver.py arg1 gwas-manhattan-plot 58
Our code relies on matplotlib for plotting, so we build using setuptools with
python setup.py bdist_egg.
The setuptools web page says:
When your project is installed (e.g., using pip), all of the dependencies not already installed will be located (via PyPI), downloaded, built (if necessary), and installed...
Within the generated pyspark_utils-0.0.1-py3.9.egg is a requires.txt file with the contents:
matplotlib
numpy
Unfortunately, when I run the spark-submit command, it can't find matplotlib:
ModuleNotFoundError: No module named 'matplotlib'
While the egg file indicates a dependency on matplotlib, it doesn't actually include the packaged code. How can I indicate to spark-submit where to find matplotlib? I thought of adding another --py-files argument for matplotlib, but matplotlib is available as a wheel, not as an egg file, and as I understand it spark-submit can't handle wheels.