Service Fabric Application Deployment Timeouts

user_15459514 0
Viewed 142

We recently upgraded our development Service Fabric cluster from version 7.1 to the latest and greatest version (8.2) and now a number of our microservice applications are failing to deploy. They are failing during 'Deploy Service Fabric Application' step in Azure DevOps at the point at which the PowerShell script is attempting to create the application with an error of 'System.TimeoutException'. By this point, the script has successfully copied the application to the image store and registered it with Service Fabric.

The 'Publish-NewServiceFabricApplication.ps1' PowerShell script operates with a two minute timeout by default and I have tried increasing the timeout by supplying a higher value via the -TimeoutSec parameter but it seems to simply ignore this. When I try to deploy the applications manually (using the New-ServiceFabricApplication PowerShell cmdlet) the deployment succeeds but it takes just over three minutes.

Has anyone come across this, is it a known issue?

azure-service-fabric azure-pipelines-release-pipeline azure-devops
1 Answers

In the end I ended up 'hacking' the 'Utilities.ps1' file (which is supplied with the Microsoft Azure Service Fabric SDK) in order to get it to respect the increased timeout values that I was supplying. But I intend to raise this issue with MS because it appears to be a bug.

Here's the relevant function New-ServiceFabricApplicationAction (line 741 of the Utilities.ps1 file) with my change included (asterisks added for emphasis):

function New-ServiceFabricApplicationAction
{
    Param (
        [string]
        $ApplicationName,

        [string]
        $ApplicationTypeName,

        [string]
        $ApplicationTypeVersion,

        [hashtable]
        $ApplicationParameter
    )

    $global:operationId = $SF_Operations.CreateNewApplication
    $createAction = {
        New-ServiceFabricApplication -ApplicationName 
        $ApplicationName -ApplicationTypeName
        $ApplicationTypeName -ApplicationTypeVersion $ApplicationTypeVersion
        ***-TimeoutSec 600*** -ApplicationParameter $ApplicationParameter
    }
    $exceptionRetryEvaluator = {
        param($ex)

        # If app already created, don't retry
        if ($ex.GetType().FullName -eq "System.Fabric.FabricElementAlreadyExistsException")
        {
            return $false
        }
        return $true
    }

    try
    {
        Invoke-ActionWithDefaultRetries -Action $createAction `
            -RetryMessage (Get-VstsLocString -Key SFSDK_RetryingCreateApplication) `
            -ExceptionRetryEvaluator $exceptionRetryEvaluator `
            -RetryableExceptions @("System.Fabric.FabricTransientException", "System.Fabric.FabricElementAlreadyExistsException", "System.TimeoutException")
    }
    catch [System.TimeoutException]
    {
        Write-Host (Get-VstsLocString -Key SFSDK_CreateApplicationFailed)
        # print application health status if create did not succeed
        Trace-ServiceFabricApplicationHealth -ApplicationName $ApplicationName
        throw
    }
}
user_154595140
Related