Network interface must specify a subnet if the network resource is in custom subnet mode

user_319538 0
Viewed 2168

Trying to run Google Dataflow examples using Python SDK.

I am able to run locally:

python -m apache_beam.examples.wordcount --output OUTPUT_FILE

However when trying to run on GCP:

python -m apache_beam.examples.wordcount \
    --project myproject \
    --job_name myproject-wordcount \
    --runner DataflowRunner \
    --staging_location gs://myproject/staging \
    --output gs://myproject/output \
    --network myproject-network \
    --zone europe-west1-b \
    --subnetwork regions/europe-west1/subnetworks/europe-west1 \
    --temp_location gs://myproject/temp

I get the following error:

apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
(9d6636d5e214c789): Workflow failed. Causes: (ab9869cb8161ec27): Error:
 Message: Invalid value for field 'resource.properties.networkInterfaces[0].subnetwork': ''. Network interface must specify a subnet if the network resource is in custom subnet mode.
 HTTP Code: 400

I am using apache-beam Python SDK 0.6.0

Can anyone help?

google-cloud-dataflow python
2 Answers

add --subnetwork with default subnetwork fixed my problem:

mvn compile exec:java \
-Dexec.mainClass=my.MainClass \
-Dexec.args="--runner=DataflowRunner \
--project=my-project \
--stagingLocation=gs://my-bucket/staging \
--gcpTemplateLocation=gs://my-bucket/template \
--templateLocation=gs://my-bucket/template \
--gcpTempLocation=gs://my-bucket/tmp \
--region=us-central1 \
--subnetwork=regions/us-central1/subnetworks/default" \
-P dataflow-runner
user_52892880

This is probably too late for you but if anyone else is getting the same problem the issue is here:

this line:
--zone europe-west1-b \
should just be:
--region europe-west1 \

user_129858780
Related