working with Hadoop Dataproc-Clusters on the google cloud platform I constantly get the message "Kernel restarting, Kernel (...) appears to have died. It will start automatically", and lose all my calculations in the jupyter notebook. I dont really see any pattern behind those errors, heavy calculations (pandas dataframes with >10M rows) are a the only driver I could identify. Cluster is set up with these configurations:
gcloud dataproc clusters create cluster-v1
--region europe-west1
--zone europe-west1-b
--subnet datascience
--no-address
--scopes 'https://www.googleapis.com/auth/cloud-platform'
--enable-component-gateway
--optional-components ANACONDA,JUPYTER
--bucket xxx-datascience
--image-version 1.5-debian10
--num-workers 3
--worker-machine-type c2-standard-16
--worker-boot-disk-size 500
--master-machine-type c2-standard-16
--master-boot-disk-size 500
--gce-pd-kms-key XXX
--project ssc
Any idea what triggers this error?