Google dataflow created via google cloud deployment manager

user_11345980 0
Viewed 425

I'm trying to create a dataflow job with template Pub/Sub to BigQuery via deployment manager. The problem is that I don't see any dataflow service after running gcloud deployment-manager types list command.

Is it possible to have an yaml file to create dataflow job?

google-cloud-dataflow google-cloud-platform
2 Answers

I'm going to provide you with a working sample of how to setup Dataflow for Cloud Deployment Manager, based on the answer from @lukaszberwid.

First you need an options.yaml file, to setup the auth (additionally you may need to setup your service account within your env):

options:
  inputMappings:
  - fieldName: Authorization
    location: HEADER
    value: '$.concat("Bearer ", $.googleOauth2AccessToken())'

Then, to register dataflow run:

gcloud beta deployment-manager type-providers create dataflow \
    --descriptor-url='https://dataflow.googleapis.com/$discovery/rest?version=v1b3' \
    --api-options-file=options.yaml

Next you can find a sample resources.yaml that can launch the WordCount template from the get started guide:

resources:
  - name: "datafow-job"
    type: "[PROJECT-ID]/dataflow:dataflow.projects.locations.templates.launch"
    properties:
      projectId: [PROJECT-ID]
      location: us-central1
      gcsPath: "gs://dataflow-templates-us-central1/latest/Word_Count"
      jobName: testing-job
      environment:
        bypassTempDirValidation: false
        tempLocation: "gs://[BUCKET-NAME]/dataflow/tmp"
        ipConfiguration: "WORKER_IP_UNSPECIFIED"
        additionalExperiments: []
      parameters:
        inputFile: "gs://dataflow-samples/shakespeare/kinglear.txt"
        output: "gs://[BUCKET-NAME]/dataflow/output/my_output"

To run it:

gcloud deployment-manager deployments create my-deployment \
    --config resources.yaml
user_26152480

If you have a RESTful CRUD API with OpenAPI specification, it can be used as a type provider. docs

Dataflow provider is not registered by default and you'll have to register that yourself. documentation

To register dataflow run:

gcloud deployment-manager type-providers create dataflow 
 --descriptor-url='https://dataflow.googleapis.com/$discovery/rest?version=v1b3'

you can see all the types avaiable for dataflow with:

gcloud deployment-manager types list --provider dataflow
user_103271890
Related