Dataproc is not installing custom Conda package from custom Conda channel

user_791226 0
Viewed 1055

I am attempting to spin up a single node Dataproc "cluster" in GCP that installs additional packages from both conda-forge and a custom Conda channel. The gcloud command I run is:

gcloud beta dataproc clusters create MY_CLUSTER_NAME \
  --enable-component-gateway \
  --bucket MY_GCS_BUCKET \
  --region us-central1 \
  --subnet default \
  --zone us-central1-a \
  --single-node \
  --master-machine-type n1-standard-4 \
  --master-boot-disk-size 500 \
  --image-version 1.5-ubuntu18 \
  --properties spark:spark.jars.packages=org.apache.spark:spark-avro_2.12:2.4.4,spark-env:spark.jars.packages=org.apache.spark:spark-avro_2.12:2.4.4 \
  --optional-components ANACONDA,JUPYTER \
  --max-idle 7200s \
  --scopes 'https://www.googleapis.com/auth/cloud-platform' \
  --project MY_PROJECT_ID \
  --metadata='CONDA_PACKAGES=pandas matplotlib seaborn scikit-learn MY_CUSTOM_PACKAGE' \
  --metadata='CONDA_CHANNELS=conda-forge https://MY_CUSTOM_CONDA_CHANNEL'

I have verified I can conda install -c https://MY_CUSOMT_CONDA_CHANNEL MY_CUSTOM_PACKAGE locally, and that other packages are being installed. When searching through the logs for the cluster, I find no entries about the installation of the additional conda packages.

Questions:

  1. Where can I find logs that will help me debug this problem?
  2. Is there something wrong with the above command?
conda google-cloud-dataproc apache-spark google-cloud-platform
1 Answers

It seems that you didn't add the conda-install.sh init action when creating the cluster, see more details in this doc, e.g.:

gcloud dataproc clusters create my-cluster \
    --image-version=1.4 \
    --region=${REGION} \
    --metadata='CONDA_PACKAGES=pandas matplotlib seaborn scikit-learn MY_CUSTOM_PACKAGE' \
    --metadata='CONDA_CHANNELS=conda-forge https://MY_CUSTOM_CONDA_CHANNEL' \
    --initialization-actions=gs://goog-dataproc-initialization-actions-${REGION}/python/conda-install.sh

You should be able to find the init action log at /var/log/dataproc-initialization-script-0.log, see more details in this doc.

user_4042640
Related