Azure Function ARM Deployment with listkeys results in BadRequest Error

Viewed 3885

I have a simple ARM template that deploys two Azure Functions, an App Service Plan and a Storage Account:

enter image description here

The only "special" thing is, that the function function-key-issue-two adds the default host key from the function function-key-issue-one as an app setting:

"FunctionOneKey": "[listkeys(concat(variables('functionTwoAppId'), '/host/default/'),'2016-08-01').functionKeys.default]",

If I deploy this template to a new resource group, it works the first time. Every subsequent deployment fails with a Bad Request Error on the Resource function-key-issue-one/default:

enter image description here

This is how the operation details looks like:

{
    "Code": "BadRequest",
    "Message": "Encountered an error (ServiceUnavailable) from host runtime.",
    "Target": null,
    "Details": [
        {
            "Message": "Encountered an error (ServiceUnavailable) from host runtime."
        },
        {
            "Code": "BadRequest"
        },
        {
            "ErrorEntity": {
                "Code": "BadRequest",
                "Message": "Encountered an error (ServiceUnavailable) from host runtime."
            }
        }
    ],
    "Innererror": null
}

If I remove the FunctionOneKey App Setting, the deployment works. Also If I don't specify the App Setting WEBSITE_RUN_FROM_PACKAGE, the deployment also works.

The function code is deployed later using the AzureFunctionApp@1 Azure DevOps Task as a Zip package (that is why I set WEBSITE_RUN_FROM_PACKAGE to 1).


How to reproduce:

The ARM template I am using is available here. You can deploy it using. e. g. the New-AzResourceGroupDeployment cmdlet:

New-AzResourceGroupDeployment -ResourceGroupName 'function-key-issue-rg' -TemplateFile "D:\sources\issues\functionDeployment\azuredeploy.json" -name "azuredeploy-$(New-Guid)"

Update 1:

The reason for the ServiceUnavailable error is probably because Kudu adds a web.config with a rewrite rule (because I use WEBSITE_RUN_FROM_PACKAGE but don't have deployed the function):

enter image description here

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name = "Site Unavailable" stopProcessing = "true">
                    <match url = ".*" />
                    <action type = "CustomResponse" statusCode = "503" subStatusCode = "0" statusReason = "Site Unavailable" statusDescription = "Could not download zip" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

My next attempt was to prevent Kudu from doing this by setting SCM_TOUCH_WEBCONFIG_AFTER_DEPLOYMENT to 0 (See: Don't touch web.config at the end of the deployment). And now it looks like, subsequent deployments sometimes succeed:

enter image description here

But still not a reliable solution :-/.


Update 2:

  • Same issue with the Azure Function Runtime ~2.
  • Switching the Azure Function to Linux also doesn't solve the issue.

Update 3:


Any idea what is wrong here? Any workarounds?

6 Answers

I too have had this problem deploying Function Apps and app settings with ARM templates.

After a back-and-forth with Azure support, we realised that the persistence of the app settings causes a restart of the Function App during deployment, which results in a 503 Service Unavailable while it spins back up. This happens while the deployment is still in progress, which then causes intermittent failures of the Microsoft.Web/sites/host. This is also despite the ARM deployment mode being set to incremental, which seems to be ignored completely for function app settings.

The Diagnostics Settings of the Function App will list the hard restarts for you and might give some insight into the app setting that caused it.

A suggestion was made from Azure support to separate the app settings into their own Microsoft.Web/sites/config section in the ARM template, which dependsOn the Function App having finished deployment. I've not tried this yet, and this also goes against what's in the Function App ARM examples, where they are a child config resource of a Microsoft.Web/sites.

I think you can change the way you are establishing communication between your functions, and you will also fix your issue. I would recommend you to use Azure Managed Identity to configure the communication between your functions, instead of using the function keys. Please have a look at this article to get more details of what I am saying.

first of all, i would suggest to use Azure KeyVault as a default storage for your keys, like described here. But it seems, that this is related also some issues regarding Appservice and Package Deployments. Take a look here: https://github.com/microsoft/azure-pipelines-tasks/issues/10961 and here: https://github.com/microsoft/azure-pipelines-tasks/issues/11444. The documentation also says something like this: https://docs.microsoft.com/de-de/azure/azure-functions/run-functions-from-deployment-package

Hope this is helpfull.

I had, sort of, the same problem in a template, that deployed EventGrid Topic subscribers to a Topic. Precense of the the listkey() function in the json template resulted in the same, very non-descriptive error message.

I've made it work, by updating the ARM template schema to the newest schema supported on Azure: 2019-08-01

Like this: Top line in the file: "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentParameters.json#",

and inline, in the listkeys function:

listkeys(concat(resourceId('Microsoft.Web/sites', parameters('functionAppName')), '/host/default'), '2019-08-01').functionKeys.default)

Now, several, subsequent deployments work again, at least for me.

I have had this problem recently and contacted the Azure Functions team, and what I understood from them is this: (I am using ARM Template should be same logic to all with different approaches)

1- using WEBSITE_RUN_FROM_PACKAGE = 1 must be set before deploying the Zip File, meaning adding dependency between the ZipDeploy with deploying AppSettings.

why? if this flag is not set at first, the function would take the ZipFile and extract it to wwroot folder, leading to empty SitePackage folder and breaking the deployment and function. this flag makes the function upload a zip file to a folder in the function shared file system under data/SitePackage, with also a package.txt that has the name of the related Zip.

this flag aren't supposed to be changed back and forth, because it would break the deployment due to confusing whether it should upload the zip to SitePackages or Extract it to wwroot.

an answer i was given if turning your flag for the first time, it could happen that the first deployment fail, and you need to try again.

2- when using WEBSITE_RUN_FROM_PACAKAGE the function go through a sort of restart, that is written on the docs here you see that after deployment a restart is done.

so the step of "listKeys" could fall on a restart and leading to failure, what you need to do is Add a "wait" step for 1-2 minutes after the function deployed to be sure that everything is done.

Good Luck

I have faced pretty much the same issue. I need to feed the Azure Function which its own host key. After several different test scenarios my conclusion is that there is some delay between the Azure Function App deployment finish and the moment the Host keys are available.

My current workaround is to create previously a key in the KeyVault a use this key in two different places:

  1. To create a new host key in the Azure Function
  2. To feed the Azure Function appsettings

Some sample codes:

Generating Key in the KeyVault from PS

$secureSecret = ConvertTo-SecureString New-Guid.ToString() -AsPlainText -Force

Set-AzureKeyVaultSecret -VaultName $keyVaultName -Name "azure-function-key" -SecretValue $secureSecret

Create the host key and passing it to the function

Note that parameters('internalKey') would be fed from the KeyVault reference

"apiVersion": "2019-08-01",
"type": "Microsoft.Web/sites",
"kind": "functionapp",
"name": "[variables('functionAppName')]",
"location": "[variables('location')]",
"resources": [{
   "dependsOn": ["[resourceId(resourceGroup().name, 'Microsoft.Web/sites', variables('functionAppName'))]"],
   "type": "Microsoft.Web/sites/host/functionKeys",
   "apiVersion": "2018-11-01",
   "name": "[concat(variables('functionAppName'), '/default/internalkey')]",
   "properties": {
      "name": "internalkey",
      "value": "[parameters('internalKey')]"
   }
},
{
   "apiVersion": "2019-08-01",
   "name": "appsettings",
   "type": "config",
   "dependsOn": [
      "[resourceId(resourceGroup().name, 'Microsoft.Web/sites', variables('functionAppName'))]"
   ],
   "properties": {
      "FUNCTIONS_EXTENSION_VERSION": "~3",
      "HostFunctionKey": "[parameters('internalKey')]"
   }
}]

Even in you try the listKeys approach with your new host key it won't get the value. First time it will fail and successive times it will get the previous value if it has changed.

Related