To speed up my cluster instantiation time, I've created a custom image with all the additional dependencies installed using miniconda3 available for dataproc image 1.5.34-debian10. (I followed the steps here: GCP Dataproc custom image Python environment to ensure I used the correct python environment).
However, when I start my cluster with --optional-components ANACONDA,JUPYTER my custom dependencies are removed and I'm left with a base installation of anaconda and jupyter. I assume the anaconda installation is overwriting my custom dependencies. Is there any way to ensure my dependencies aren't overwritten? If not, is it possible to install anaconda and jupyter as part of my custom dataproc image instead?
I've used the following command to create the custom image:
python3 generate_custom_image.py \
--image-name test-dataproc-image-1 \
--dataproc-version 1.5.34-debian10 \
--customization-script ./pip-install.sh \
--zone <my_zone> \
--gcs-bucket <my_gcs_bucket> \
--subnet <my_subnet> \
--metadata 'PIP_PACKAGES="dask[distributed] dask-yarn==0.8 google-cloud-pubsub==2.4.1 kneed==0.7.0 google-cloud-secret-manager==2.4.0 pandas-gbq==0.15.0 pyarrow==3.0.0 leidenalg==0.8.4 cdlib==0.2.0 bigquery==0.0.12 google-cloud-storage==1.37.1 facebook_business pyemd==0.5.1 pysftp==0.2.9 gensim==4.0.1 wordcloud==1.8.1 spacy==2.3.0 https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz --ignore-installed PyYAML"'
--no-smoke-test
Below is the content of my custom script pip-install.sh :
which python
# Activate miniconda3 optional component.
cat >>/etc/google-dataproc/dataproc.properties <<EOF
dataproc.components.activate=miniconda3
EOF
export BDUTIL_DIR=/usr/local/share/google/dataproc/bdutil
bash /usr/local/share/google/dataproc/bdutil/components/activate/miniconda3.sh
source /etc/profile.d/effective-python.sh
# Now this is /opt/conda/default/bin/python
which python
apt update
apt install python3-pip -y
python -m pip install --upgrade ${PACKAGES}