Azure datafactory: My REST api call works in copy data but not in data flow, why?

flyingDutchman 0
Viewed 74

I am trying to set up a data pipeline in ADF. I can't seem to figure out why an API call works as a source in the "copy data" block in the pipeline, but when I try to use it as a source in the data flow it throws an error.

This is the request in the copy data block: https://i.stack.imgur.com/9wzSZ.png'

This is the request in the data flow tab: : https://i.stack.imgur.com/Onpuk.png

And this is the error I get: https://i.stack.imgur.com/f9zdb.png

The request body should be the same. Still I get a good response when I preview the data in the copy block: https://i.stack.imgur.com/cKZsb.png

Is there some setting that is different that I am missing?

EDIT: the url for the request is https://www.daggegevens.knmi.nl/klimatologie/uurgegevens with the request body start=2022010101&end=2022090724&stns=330&vars=DD:FH:FF:FX&fmt=json

azure-data-factory-pipeline azure-data-factory azure-data-factory-2 azure
1 Answers

I have reproduced same error in Dataflow.

error

  • I tried to give Response body in query parameters as in below image

Query params

  • Response of this API had header and body in each line item. response
  • Thus, select transformation is used to select only the body response and copied it to the sink select transformation

Outputs that I received from both copy activity and data flow activity are same.

copy activity output copy activity output

data flow activity outputdata flow activity output

Aswin0
Related