How to parallelize loops with an include in ansible

user_2172072 0
Viewed 7131

Recently I came across a bottleneck in our ansible playbooks' code. We were deploying our clusters (e.g. a mongoDB Replica Set) sequentially - i.e. one VM after another, each waiting for the previous to be up and running.

This slowed down the whole cluster deploy time by a factor of the members on it.

To solve this, I started digging on ansible's async actions and pooling and found out a few examples on parallel loops and "fire-and-forget" strategies for scenarios like ours.

The particular thing is, we have defined our own "customize the VM and spawn it" ansible task (create_instance.yml) that gets included and receives the different customization variables from the playbook and abstracts the whole process by running different KVM/shell commands.

Using "Parallel task execution in Ansible" as reference, I ended up having something like: 

- name: Generate VMs for DB
  hosts: hypervisor_fe
  tags: platform,mongodb
  tasks:
    - include: tasks/create_instance.yml
      vars:
        vm: "{{ item }}"
      with_items: "{{ mongodb.vms }}"
      register: mongo_instances
      async: 7200
      poll: 0

    - name: Wait for instance creation to complete
      async_status: jid={{ item.ansible_job_id }}
      register: mongo_jobs
      until: mongo_jobs.finished
      retries: 300
      with_items: "{{ mongo_instances.results }}"

However, this setup does seem to ignore all the new async code and keeps the old, sequential behavior. I'm guessing this has to do with the no. and granularity of plays inside the imported task. If I instead replace the include for a single, explicit long-running task - let's say, e.g.

- name: Test async operation
  shell: ping -c1 {{ item.hostname }} && sleep 20

This does seem to work just fine, running one ping to each item and then moving on to the next action.

Is this assumption right? Does someone has experience with include and async loops in ansible? Do I need to move the async declaration to a single play inside the imported code?

ansible asynchronous
1 Answers

I advise you to rethink your playbook design in the following way:

- hosts: localhost
  tasks:
    - add_host:
        name: "{{ item.name }}"
        groups: new_vms
        vm: "{{ item }}"
      with_items: "{{ mongodb.vms }}"

- hosts: new_vms
  tasks:
    - include: create_instance.yml

And inside create_instance.yml use delegate_to: hypervisor_fe.

This gives you native Ansible host loop for every vm with concurrent execution of each task.

user_27955920
Related