Unable to run pySpark job on the Dataproc

Viewed 243

I try to run pySpark job on the new Dataproc cluster created using:

gcloud beta dataproc clusters create ${CLUSTER_NAME} \
 --region=${REGION} \
 --image-version=1.4 \
 --master-machine-type=n1-standard-4 \
 --worker-machine-type=n1-standard-4 \
 --bucket=${BUCKET_NAME} \
 --optional-components=ANACONDA,JUPYTER \
 --enable-component-gateway

then I try to run different jobs, all that create spark context result in infinite loop:

 WARN org.apache.spark.scheduler.cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

example script:

import pyspark

sc = pyspark.SparkContext()
rdd = sc.parallelize(['Hello,', 'world!', 'dog', 'elephant', 'panther'])
words = sorted(rdd.collect())
print(words)

added to the cluster using

gcloud dataproc jobs submit pyspark pyspark_sort.py \
    --cluster=${BUCKET_NAME} \
    --region=us-central1

When I run something like that:

import getpass
import sys
import imp

print('This job is running as "{}".'.format(getpass.getuser()))
print(sys.executable, sys.version_info)
for package in sys.argv[1:]:
    print(imp.find_module(package))

I get a successful output:

Job [f10a70ab95c54c8a9fa471429b91153b] submitted.
Waiting for job output...
This job is running as "root".
/opt/conda/default/bin/python sys.version_info(major=3, minor=6, micro=10, releaselevel='final', serial=0)
(None, '/opt/conda/default/lib/python3.6/site-packages/pandas', ('', '', 5))
(None, '/opt/conda/default/lib/python3.6/site-packages/scipy', ('', '', 5))
Job [f10a70ab95c54c8a9fa471429b91153b] finished successfully.

Thank's in advance!

0 Answers
Related