rollback helm release on helm test failure

Question

Let's say I've release that has test suite associated with it.

So the typical installation would look like:

helm upgrade --install service service/

and shortly after:

$ helm test service-test
NAME: service
LAST DEPLOYED: Thu Jul 15 15:45:40 2021
NAMESPACE: default
STATUS: deployed
REVISION: 4
TEST SUITE:     service-test
Last Started:   Thu Jul 15 15:45:45 2021
Last Completed: Thu Jul 15 15:46:00 2021
Phase:          Succeeded

This is how happy path for test suite looks like.

But let's think of less happy scenario:

$ helm test service-test
NAME: service
LAST DEPLOYED: Thu Jul 15 15:45:40 2021
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE:     service-test
Last Started:   Thu Jul 15 15:25:48 2021
Last Completed: Thu Jul 15 15:26:54 2021
Phase:          Failed

So there's clear indication of failure and 'Failed' substring can be looked up to trigger helm rollback service 0 thereafter, but this approach looks weird to me.

How do I properly rollback on a failed test suite with helm built-in mechanism or some other tool that does not involve piping helm test command output into sed/awk?

user_3441253 · Answer

Short answer is: helm hooks.

The way it's currently designed in helm release management, test got to have annotation for helm to be able to recognise terminal status of release.

Furthermore, there's special mechanism to trigger rollback in-place if test has failed. Keep in mind that all resources associated with test suite are cleaned up in this case so helm history got to be monitored for actual release result (or remove hook-failed to debug it).

The solution:

apiVersion: batch/v1
kind: Job
metadata:
  name: "{{ .Release.Name }}-test"
  labels:
    app: {{ .Release.Name }}
    release: {{ .Release.Name }}
  annotations:
    "helm.sh/hook": test-success, post-upgrade
    "helm.sh/hook-delete-policy": hook-succeeded, hook-failed
    "helm.sh/hook-weight": "1"

If I getting it correctly, this annotations translating to the following:

run this test after upgrade (i.e. pod is up and running)
monitor status of test
fail release and rollback if test indicates failure
if test has no failure indication then all went smooth, no further actions required
regardless of test result clean up all resources that have association with the test suite (1 job & 2 pods in my case)

rollback helm release on helm test failure

1 Answers