Google Cloud Storage: Output path does not exist or is not writeable

user_3549711 0
Viewed 6933

I am trying to follow this simple Dataflow example from google cloud site.

I have successfully installed the dataflow pipeline plugin and gcloud SDK (as well as Python 2.7). I have also set up a project on google cloud and enabled billing and all the necessary API's - as specified in the instructions above.

However, when I go to the run configurations and change the Pipeline Arguments tab to select BlockingDataflowPipelineRunner, after entering creating a bucket and setting my project-id, hitting run gives me:

Caused by: java.lang.IllegalArgumentException: Output path does not exist or is not writeable: gs://my-cloud-dataflow-bucket
    at com.google.cloud.dataflow.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
    at com.google.cloud.dataflow.sdk.util.DataflowPathValidator.verifyPathIsAccessible(DataflowPathValidator.java:79)
    at com.google.cloud.dataflow.sdk.util.DataflowPathValidator.validateOutputFilePrefixSupported(DataflowPathValidator.java:62)
    at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.fromOptions(DataflowPipelineRunner.java:255)
    at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.fromOptions(BlockingDataflowPipelineRunner.java:82)
    ... 9 more

I have used my terminal to execute 'gcloud auth login' and I see in the browser that I am successfully logged in.

I am really not sure what I have done wrong here. Can anyone confirm if this is a known issue with using dataflow pipeline and google buckets?

Thanks!

gsutil google-cloud-dataflow google-cloud-storage jenkins permissions
5 Answers

I had a similar issue with GCS bucket permissions, though I certainly had write permissions and I could upload files into the bucket. What solved the problem for me was acquiring roles/dataflow.admin permission for the project I was submitting the pipeline to.

user_25094650

Try to provide zone parameter, it works in my case with similar error. And of course export GOOGLE_APPLICATION_CREDENTIALS environment variable before running your app.

 ...
 -Dexec.args="--runner=DataflowRunner \
 --gcpTempLocation=gs://bucket/tmp \
 --zone=bucket-zone \
 ...
user_73215570

Got the same error. Fixed it by setting GOOGLE_APPLICATION_CREDENTIALS using the key file with write permissions in ~/.bash_profile on Mac.

user_78288130
Related