Best way to check app health endpoint after deployment in a YAML deployment job?

My solution was to create a fault-tolerant PowerShell script to ping the health endpoint. I took this a step further and wrapped it in a template which can be inserted into your main CI/CD template. Doing this lets me develop and test the PS script in isolation with my IDE of choice (VSCode) and also re-use it as a standard task thanks to the template, or use it outside of Azure Pipelines.

Script

Notes on script: The script ultimately decides if the task fails or not via Write-Error combined with the failOnStderr: true for PowerShell@2 task we will see later.

## Base code borrowed from https://karask.com/retry-powershell-invoke-webrequest/
## Slightly modified to remove things like the file logging.

param (
    [string]$URI,
    [string]$Method = 'GET',
    [string]$SuccessTextContent = 'Healthy',
    [string]$Retries = 1,
    [string]$SecondsDelay = 2,
    [string]$TimeoutSec = 120
)

Write-Output "$Method ""$URI"" Retries: $Retries, SecondsDelay $SecondsDelay, TimeoutSec $TimeoutSec";

Function Req {
    Param(
        [Parameter(Mandatory=$True)]
        [hashtable]$Params,
        [int]$Retries = 1,
        [int]$SecondsDelay = 2
    )

    $Params.Add('UserAgent', 'azagent powershell task')

    $method = $Params['Method']
    $url = $Params['Uri']

    $cmd = { Write-Host "$method $url..." -NoNewline; Invoke-WebRequest @Params }

    $retryCount = 0
    $completed = $false
    $response = $null

    while (-not $completed) {
        try {
            $response = Invoke-Command $cmd -ArgumentList $Params
            if ($response.StatusCode -ne 200) {
                throw "Expecting reponse code 200, was: $($response.StatusCode)"
            }
            $completed = $true
        } catch {
            Write-Output "$(Get-Date -Format G): Request to $url failed. $_"
            if ($retrycount -ge $Retries) {
                Write-Error "Request to $url failed the maximum number of $retryCount times."
                throw
            } else {
                Write-Warning "Request to $url failed. Retrying in $SecondsDelay seconds."
                Start-Sleep $SecondsDelay
                $retrycount++
            }
        }
    }

    Write-Host "OK ($($response.StatusCode))"
    return $response
}

$res = Req -Retries $Retries -SecondsDelay $SecondsDelay -Params @{ 'Method'=$Method;'Uri'=$URI;'TimeoutSec'=$TimeoutSec;'UseBasicParsing'=$true }

if($res.Content -ne "$SuccessTextContent")
{
    Write-Error $response.Content
}
else
{
    Write-Host "Helath check validation success."
}

Template

Notes on Template: There is a subtle detail about this task that is easy to miss. - checkout: templates. This will actually check out the repository resource defined in the template in which this template is inserted into. Which seems kind of obvious since the template is inserted and then once its finished its executed as if it were one yaml.

parameters:
  - name: URI
    type: string
  - name: Method
    type: string
    default: 'GET'
  - name: SuccessTextContent
    type: string
    default: 'Healthy'
  - name: Retries
    type: number
    default: 5
  - name: SecondsDelay
    type: number
    default: 2
  - name: TimeoutSec
    type: number
    default: 120

steps:
  - checkout: templates
  - task: PowerShell@2
    displayName: '${{ parameters.Method }} ${{ parameters.URI }}...'    
    inputs:
      failOnStderr: true
      targetType: 'filePath'
      filePath: $(System.DefaultWorkingDirectory)\InvokeRequestWithRetry.ps1
      arguments: > # Use this to avoid newline characters in multi-line string
        -URI "${{ parameters.URI }}"
        -Method "${{ parameters.Method }}"
        -SuccessTextContent "${{ parameters.SuccessTextContent }}"
        -Retries ${{ parameters.Retries }}
        -SecondsDelay ${{ parameters.SecondsDelay }}
        -TimeoutSec ${{ parameters.TimeoutSec }}

Parent CI/CD YAML

Notes on Parent YAML: Finally, we have the usage. I chose to make this its own dedicated deployment job so I can manually click retry if everything fails and not have to re-run the whole deployment. Also, I wanted the PS to run against my VM environments behind my firewall.

resources:
  repositories:
    - repository: templates
      type: git
      name: IAMM/azure-pipelines-templates
      ref: refs/tags/v0.4

## ...
## Removed for brevity
## ...

- deployment: MyAppHealthDevIIS
  dependsOn: MyAppDeployDevIIS
  displayName: 'Hit /health/views endpoint before proceeding'
  environment:
    name: Development
    resourceType: VirtualMachine
  strategy:
    runOnce:
      deploy:
        steps:
        - template: templates/health-check.yaml@templates
          parameters:
            URI: 'https://$(iisHostName)/$(iisTargetApplication)/health/views'
            SecondsDelay: 5