I've created a custom connector that uses authorization flow to connect to third-party APIs and I use an enterprise gateway to schedule the refresh for the dataset. The issue with that is, that the old dataset is being replaced by the new dataset after every refresh. I have scheduled the refresh to run every 1 hour. So at the end of the day, I lose all the reports. So as a solution for this, I created a push dataset which I believe is backed up by a DB, and using the rest API I'm pushing the refreshed data to the push dataset. Below is the code for that.
pushdataset = (data) =>
let
headers = [RelativePath = "https://api.powerbi.com", IsRetry=true, Headers = [
#"Content-Type"="application/json", Accept="application/json"
], Content = Json.FromValue(data)],
response = Web.Contents("/beta/77777/datasets/66789900/rows?key=ccccc", headers),
in
response;
When I execute the connector I get "Access is forbidden 403" error. It seems like a simple HTTP request, I can access the dataset using the python code and also from the postman.
I'm stuck with this for a long time. How do we connect to the push dataset from the custom connector? Also if there are other ways to keep the existing data and append new data to the dataset after every refresh, please let me know.