I try to run pySpark job on the new Dataproc cluster created using:
gcloud beta dataproc clusters create ${CLUSTER_NAME} \
--region=${REGION} \
--image-version=1.4 \
--master-machine-type=n1-standard-4 \
--worker-machine-type=n1-standard-4 \
--bucket=${BUCKET_NAME} \
--optional-components=ANACONDA,JUPYTER \
--enable-component-gateway
then I try to run different jobs, all that create spark context result in infinite loop:
WARN org.apache.spark.scheduler.cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
example script:
import pyspark
sc = pyspark.SparkContext()
rdd = sc.parallelize(['Hello,', 'world!', 'dog', 'elephant', 'panther'])
words = sorted(rdd.collect())
print(words)
added to the cluster using
gcloud dataproc jobs submit pyspark pyspark_sort.py \
--cluster=${BUCKET_NAME} \
--region=us-central1
When I run something like that:
import getpass
import sys
import imp
print('This job is running as "{}".'.format(getpass.getuser()))
print(sys.executable, sys.version_info)
for package in sys.argv[1:]:
print(imp.find_module(package))
I get a successful output:
Job [f10a70ab95c54c8a9fa471429b91153b] submitted.
Waiting for job output...
This job is running as "root".
/opt/conda/default/bin/python sys.version_info(major=3, minor=6, micro=10, releaselevel='final', serial=0)
(None, '/opt/conda/default/lib/python3.6/site-packages/pandas', ('', '', 5))
(None, '/opt/conda/default/lib/python3.6/site-packages/scipy', ('', '', 5))
Job [f10a70ab95c54c8a9fa471429b91153b] finished successfully.
Thank's in advance!