Running HDFS commands on a dataproc cluster if you can't SSH onto the master node

Question

Is there a way of submitting HDFS commands on a Dataproc cluster, if you can't SSH into the master node. I couldn't find anything in the Gcloud SDK or the rest API.

So something like :

gcloud dataproc jobs submit hdfs 'ls /hdfs/path/' --cluster=my-cluster --region=europe-west1

user_3777211 · Answer

The easiest way is pig fs via gcloud dataproc jobs submit pig --execute 'fs -ls /', or otherwise pig sh as a catch-all for other shell commands.

Running HDFS commands on a dataproc cluster if you can't SSH onto the master node

1 Answers