Ansible: How to find aggregated file size across inventory hosts?

userfRKK 0
Viewed 43

I'm able to find the total size of all the three files in variable totalsize on a single host as shown below.

cat all.hosts

[destnode]
myhost1
myhost2
myhost3

cat myplay.yml

- name: "Play 1"
  hosts: "destnode"
  gather_facts: false

  tasks:
    - name: Fail if file size is greater than 2GB
      include_tasks: "{{ playbook_dir }}/checkfilesize.yml"
      with_items:
        - "{{ source_file_new.splitlines() }}"

cat checkfilesize.yml

- name: Check file size
  stat:
    path: "{{ item }}"
  register: file_size

- set_fact:
    totalsize: "{{ totalsize | default(0) |int + ( file_size.stat.size / 1024 / 1024 ) | int }}"

- debug:
    msg: "TOTALSIZE: {{ totalsize }}"

To run:

ansible-playbook -i all.hosts myplay.yml -e source_file_new="/tmp/file1.log\n/tmp/file1.log\n/tmp/file1.log"

The above play works fine and gets me the total sum of sizes of all the files mentioned in variable source_file_new on individual hosts.

My requirement is to get the total size of all the files from all the three(or more) hosts mention is destnode group.

So, if each file is 10 MB on each host, the current playbook prints 10+10+10=30MB on host1 and like wise on host2 and host3.

Instead, I wish to the the sum of all the sizes from all the hosts like below

host1 (10+10+10) + host2 (10+10+10) + host3 (10+10+10) = 90MB

subtotal ansible-facts ansible
2 Answers

Extract the totalsize facts for each node in destnode from hostvars and sum them up.

In a nutshell, at the end of your current checkfilesize.yml task file, replace the debug task:

- name: Show total size for all nodes
  vars:
    overall_size: "{{ groups['destnode'] | map('extract', hostvars, 'totalsize') 
      | map('int') | sum }}"
  debug:
    msg: "Total size for all nodes: {{ overall_size }}"
  run_once: true

If you need to reuse that value later, you can store it at once in a fact that will be set with the same value for all hosts:

- name: Set overall size as fact for all hosts
  set_fact:
    overall_size: "{{ groups['destnode'] | map('extract', hostvars, 'totalsize') 
      | map('int') | sum }}"
  run_once: true

- name: Show the overall size (on result with same value for each host)
  debug:
    msg: "Total size for all nodes: {{ overall_size }} - (from {{ inventory_hostname }})"

As an alternative, you can replace set_fact with a variable declaration at play level.

Zeitounator0

It seems you are trying to implement (distributed) programming paradigms which aren't plain possible, at least not in that way and since Ansible is not a programming language or something for distributed computing but a Configuration Management Tool in which you declare a state. Therefore those are not recommended and should probably avoided.

Since your use case looks for me like in a normal MapReduce environment I understand from your description that you like to implement a kind of Reducer in a Distributed Environment in Ansible.

You made already the observation that the facts are distributed over your hosts in your environment. To sum them up it will be necessary that they become aggregated on one of the hosts, probably the Control Node.

To do so:

  • It might be possible to use Delegating facts for your task set_fact to get all necessary information to sum up onto one host

  • An other approach could be to let your task creating and adding custom facts about the summed up filesize during run. Those Custom Facts could become gathered and cached on the Control Node during next run.

  • A third option and since Custom Facts can be simple files, one could probably create a simple cronjob which creates the necessary .fact file with requested information (filesize, etc.) on a scheduled base.

Further Documentation

Similar Q&A

Summary

My requirement is to get the total size of all the files from all the three (or more) hosts ...

Instead of creating a playbook which is generating and calculating values (facts) during execution time it is recommended to define something for the Target Nodes and create a playbook which is just collecting the facts in question.

In example

... add dynamic facts by adding executable scripts to facts.d. For example, you can add a list of all users on a host to your facts by creating and running a script in facts.d.

which can also be about files and the size.

U880D0
Related