Automated publish Data Factory generate ARM template

Question

For some reason the NPM package creates an "invalid" arm template. Not the one that I expected. The arm template has no pipelines datasets etc.

What could be the issue here? I have a publish_config file and the package.json file in the root folder of my collaboration branche. The build pipeline is in a shared branche and its triggered from the main branche to release to dev, test and prod.

Hope I can get some help.

Package.json

{
    "scripts":{
        "build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
    },
    "dependencies":{
        "@microsoft/azure-data-factory-utilities":"^0.1.3"
    }
}

Pipeline:

trigger:
#- master 
feature/npmv2 branch

pool:
  vmImage: 'ubuntu-latest'

steps:


- task: NodeTool@0
  inputs:
    versionSpec: '10.x'
  displayName: 'Install Node.js'

- task: Npm@1
  inputs:
    command: 'install'
    verbose: true
  displayName: 'Install npm package'



- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build validate $(Build.Repository.LocalPath) /subscriptions/xxxxx/resourceGroups/xxxx/providers/Microsoft.DataFactory/factories/df-dev-0001'
  displayName: 'Validate'

# Validate and then generate the ARM template into the destination folder. Same as clicking "Publish" from UX
# The ARM template generated is not published to the ‘Live’ version of the factory. Deployment should be done using a CI/CD pipeline. 

- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build export $(Build.Repository.LocalPath) /subscriptions/xxxxx/resourceGroups/xxxx/providers/Microsoft.DataFactory/factories/df-dev-0001 "ArmTemplate"'
  displayName: 'Validate and Generate ARM template'

# Publish the Artifact to be used as a source for a release pipeline

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.Repository.LocalPath)/ArmTemplate'
    artifact: 'ArmTemplates'
    publishLocation: 'pipeline'

Publish config

{"publishBranch":"factory/adf_publish","includeFactoryTemplate":true}

the output of running run build export ... step is:

The folder structure of the one that is failing:

The folder structure of the one that is succesfull:

Kind regards,

Dickkieee

user_9680196 · Answer

The logs indicate that the needed ADF json artefacts are not present in the folder from which you are running the build command.

your npm build package.json should be in the same folder as your ADF json artefacts, you can put it in a subfolder (like build for instance)

it seems that the branch from which you are running the build pipeline does not have the latest ADF code or the code is not in the location referenced by your yamp script step ($(Build.Repository.LocalPath): root of the source code in your case)

review your branching strategy, make sure the feature branch or the code branch that trigger the build procedure contain both the ADF json artefacts (the code) and the npm package.json.

Check out my article on how to you setup your dev env correctly for using ADF utilities: http://www.mutazag.com/blog/code/tutorial/adf-utils-for-publishing/

to use the npm build command from inside a build sub folder you need to change the export and validate steps in the yaml file to refer to the correct working directory as follows:

    - task: Npm@1
      inputs:
        command: 'custom'
        workingDir: '$(Build.Repository.LocalPath)/build' #replace with the package.json folder
        customCommand: 'run build export $(Build.Repository.LocalPath) "/subscriptions/xxxxxx/resourceGroups/xxxxx/providers/Microsoft.DataFactory/factories/df-dev-mutaz" "ArmTemplate"'
      displayName: 'Export and Generate ARM template'

Automated publish Data Factory generate ARM template

1 Answers