How to run JUnit tests on multiple Ansible assertions

Shaul Behr 0
Viewed 41

I've got a GitLab pipeline that runs some Ansible playbooks in a container. At the end, I want to run some tests to ensure that everything is properly configured, e.g.

- name: Get the Frobzy value
  raw: "frobzy -x foobar"
  register: frobzy_val
  changed_when: false

- name: Check that Frobzy is valid
  assert:
    that:
      - frobzy_val.stdout is match "oogah"
    fail_msg: "bad frobzy"
  ignore_errors: true # see comments

- name: Get the Blobzy value
  raw: "blobzy -y barfoo"
  register: blobzy_val
  changed_when: false

- name: Check that Blobzy is valid
  assert:
    that:
      - blobzy_val.stdout is match "warra"
    fail_msg: "bad blobzy"
  ignore_errors: true # see comments

This is all nicely configured so that I get a JUnit report that gets displayed on my GitLab pipeline results.

The problem is, let's say both of these tests fail. If I look at the test results from my pipeline, it correctly displays that the tests failed. Yet the pipeline itself is marked as having passed! If I hadn't taken a detailed look at the test results, I would have assumed that everything is tickety-boo, and merged some potentially broken code.

The reason why the pipeline is marked as successful is because of those ignore_errors: true settings. It registers an error, but follows the directive that this should not fail the pipeline. But if I set ignore_errors: false (which is the default behavior), then as soon as the Frobzy test fails, the pipeline will abort and show a failure on the Frobzy test, but since it never ran the Blobzy test, I don't know that Blobzy would have failed, too.

Basically, I want a way to run all my Ansible assertions, even if some of them fail; I want to see all the test results in my pipeline, and I want the pipeline to fail if any tests failed.

How can I do this?

junit gitlab-ci ansible
2 Answers

  • Here's a quick solution that might help.

  • Have a task to check if any of the tasks failed. Set a failed variable to true in that case.

  • Fail the playbook at the end if this variable is set to true.


- hosts: localhost
  vars:
    failed: false
  gather_facts: no
  tasks:

  - name: Get the Frobzy value
    raw: "echo frobzy"
    register: frobzy_val
    changed_when: false

  - name: Check that Frobzy is valid
    assert:
      that:
        - frobzy_val.stdout is match "frobzy"
      fail_msg: "bad frobzy"
    ignore_errors: true # see comments

  - name: Get the Blobzy value
    raw: "echo blobzy"
    register: blobzy_val
    changed_when: false

  - name: Check that Blobzy is valid
    assert:
      that:
        - blobzy_val.stdout is match "blobzy"
      fail_msg: "bad blobzy"
    ignore_errors: true # see comments


  - name: Set a variable to true if the task fails
    set_fact:
      failed: true
    ignore_errors: true
    when: not ("'blobzy' in blobzy_val.stdout" ) or 
          not ("'frobzy' in frobzy_val.msg")


  - name: Fail if failed is true
    fail:
      msg: "Failed"
    when: failed
  • Run
ā”€ ansible-playbook test.yaml 

PLAY [localhost] ********************************************************************************************************************************************************************************

TASK [Get the Frobzy value] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [Check that Frobzy is valid] ***************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Get the Blobzy value] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [Check that Blobzy is valid] ***************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Set a variable to true if the task fails] *************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail if failed is true] *******************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP **************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
codeaprendiz0
  • Make a single assertion for both conditions and delay it until the end so that it fails after the two tests are done.
  • Add a debug task to show results for your pipelines.
- name: Get the Frobzy value
  raw: "frobzy -x foobar"
  register: frobzy_val
  changed_when: false

- name: Get the Blobzy value
  raw: "blobzy -y barfoo"
  register: blobzy_val
  changed_when: false

- name: Show results for Frobzy and Blobzy
  debug:
    msg:
      - "Frobzy is {{ 'ok' if frobzy_val.stdout is match 'oogah' else 'bad' }}"
      - "Blobzy is {{ 'ok' if lobzy_val.stdout is match 'warra' else 'bad' }}"
  
- name: Fail if any test is bad
  assert:
    that:
      - frobzy_val.stdout is match "oogah"
      - blobzy_val.stdout is match "warra"
    fail_msg: "At least one of Frobzy or Blobzy has bad results. See debug above"
``
Zeitounator0
Related