How to run a Notebook on dataproc using composer

user_1899010 0
Viewed 372

I have a Jupyter notebook on dataproc which is using dataproc for processing. I have also stored this notebook on gcs at a location "gc:/bucket/...".

I'd like to create a Composer workflow to run this notebook on my existing dataproc cluster. Is that possible in GCP?

google-cloud-composer google-cloud-dataproc google-cloud-platform
1 Answers

You cannot run a Jupyter notebook programatically in Dataproc. An approach that you can take is to extract the code from the notebook and then submit it as a job in your cluster from Composer, for example, using the DataprocPySparkOperator.

user_75177570
Related