Azure Data Factory - Use parameter for typeProperties in storage event trigger

Question

I'm having issues using parameter within a storage event trigger for the typeProperty "blobPathBeginsWith". Per default when using a storage event trigger, the typeProperty "scope" appears in the ARMTemplateParamtersForFactory.json and can be correctly set in a CI/CD process for different environments.

However, as I use the standard integration "Export to datalake" from Power Apps to Data Lake, the container name in the Data Lake is different (and cannot be changed) depending on the environment. E.g.

Environment	ContainerName
dev	dataverse-researchdwhd-xxx1
test	dataverse-researchdwhd-xxx2

Now, when I create a storage event trigger and manually fill out all required information including subscription, storage account name, container name, blob path begins with and blob path ends with, following typeProperties are created automatically:

        "typeProperties": {
            "blobPathBeginsWith": "/dataverse-researchdwhd-xxx1/blobs/apss_project/Snapshot",
            "blobPathEndsWith": ".csv",
            "ignoreEmptyBlobs": true,
            "scope": "/subscriptions/6fxxxb5a/resourceGroups/rdwh-dev/providers/Microsoft.Storage/storageAccounts/datalakerdwhdev",
            "events": [
                "Microsoft.Storage.BlobCreated"
            ]
        }

Once the trigger is published, following parameter is available in the ARMTemplateParametersForFactory.json and can therefore be set in the release pipeline.

        "trigger_snapshot_project_properties_typeProperties_scope": {
            "value": "/subscriptions/6fxxxb5a/resourceGroups/rdwh-dev/providers/Microsoft.Storage/storageAccounts/datalakerdwhdev"
        }

In my use case, not only the typeProperty "scope" is environment dependent but also the typeProperty "blobPathBeginsWith" since the auto-created container by the "Export to data lake" integration has a unique name across all environments. Therefore, I must somehow be able to parameterize the typeProperty "scope" so it can be set in a release pipeline depending on the environment it is deployed.

What I tried so far:

Created a global paramter called "container_name" and tried to manually updated the trigger json to use this global parameter.

"blobPathBeginsWith": "parameters('container_name')",

However, regardless if the paramter only contains the container name (/dataverse-researchdwhd-xxx1/) or the whole begins with path (/dataverse-researchdwhd-xxx1/blobs/apss_project/Snapshot/), once I saved the json and opened the trigger in the UI, the message "The container name is not written in an accepted format" appears below the container name dropdown.

The format should be correct based on the Microsoft doc "Examples of storage event triggers" (https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger) but it seems global parameter cannot be referenced within a trigger.

Any experts out there can lead me to the correct way to parameterize typeProperties within the trigger json besides "scope"?

Thanks in advance!

user_15519970 · Answer

The use case is describe in the Microsoft documentation, I just didn't look at the correct place: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters

On the Azure Data Factory where GIT is enabled, you can navigate to Manage > ARM template > Edit parameter configuration.

This opens arm-template-parameters-definition.json where you can add properties which are not paramtererized by default. For my use case, I added the parameter "blobPathBeginsWith" as "typeProperties" for triggers:

"Microsoft.DataFactory/factories/triggers": {
    "properties": {
        "pipelines": [
            {
                "parameters": {
                    "*": "="
                }
            },
            "pipelineReference.referenceName"
        ],
        "pipeline": {
            "parameters": {
                "*": "="
            }
        },
        "typeProperties": {
            "scope": "=",
            "blobPathBeginsWith": "="
        }
    }
}

After the changes were published, this automatically updated the file "ARMTemplateParametersForFactory.json" in the adf_publish-branch and added for all triggers a new parameter analog following pattern which can then be used in the release pipeline.

        "trigger_name_properties_typeProperties_blobPathBeginsWith": {
        "value": "/container/blobs/folder/Snapshot"
        }

Azure Data Factory - Use parameter for typeProperties in storage event trigger

1 Answers