Process Substitution in Ansible for Path-based Parameters

Disclaimer:

• I am (obviously!) the author of the below collection which was created as a reaction to the above question
• The lookup plugin was not thoroughly tested and might fail with particular modules.

Since this was a pretty good idea and nothing existed, I decided to give it a try. This all ended up in a collection now called thoteam.var_as_file which is available in a github repo. I won't paste all files in this answer as they are all available in the mentioned repo with a full README documentation to install, test and use.

The global idea was the following:

• Create a lookup plugin responsible for pushing new temporary files with a given content and returning a path to use them.
• Clean-up the created files at the end of playbook run. For this step, I created a callback plugin which launches the cleanup action listening to v2_playbook_on_stats events.

I still have some concerns about concurrency (files yet to be cleaned are stored in a static json file on disk) and reliability (not sure that the stats stage happens in all situation, especially on crashes). I'm also not entirely sure using a callback for this is a good practice / best choice.

Meanwhile this was quite fun to code and it does the job. I will see if this work is used by other and might very well enhance all this in the next weeks (and if you have PRs to fix the already know issues, I'm happy to accept them).

Once installed and the callback plugin enabled (see https://github.com/ansible-ThoTeam/thoteam.var_as_file#installing-the-collection), the lookup can be used anywhere to get a file path containing the passed content. For example:

- name: Get a filename with the given content for later use
  ansible.builtin.set_fact:
    my_tmp_file: "{{ lookup('thoteam.var_as_file.var_as_file', some_variable) }}"
    
- name: Use in place in a module where a file is mandatory and you have the content in a var
  community.general.java_cert:
    pkcs12_path: "{{ lookup('thoteam.var_as_file.var_as_file', pkcs12_store_from_vault) }}"
    cert_alias: default
    keystore_path: /path/to/my/keystore.jks
    keystore_pass: changeit
    keystore_create: yes
    state: present

These are the relevant parts of the two plugin files. I removed the ansible documentation vars (for conciseness) which you can find in the git repo directly if your wish.

plugins/lookup/var_as_file.py

from ansible.errors import AnsibleError
from ansible.plugins.lookup import LookupBase
from ansible.module_utils.common.text.converters import to_native
from ansible_collections.thoteam.var_as_file.plugins.module_utils.var_as_file import VAR_AS_FILE_TRACK_FILE
from hashlib import sha256
import tempfile
import json
import os

def _hash_content(content):
    """
    Returns the hex digest of the sha256 sum of content
    """
    return sha256(content.encode()).hexdigest()

class LookupModule(LookupBase):

    created_files = dict()

    def _load_created(self):
        if os.path.exists(VAR_AS_FILE_TRACK_FILE):
            with open(VAR_AS_FILE_TRACK_FILE, 'r') as jfp:
                self.created_files = json.load(jfp)

    def _store_created(self):
        """
        serialize the created files as json in tracking file
        """

        with open(VAR_AS_FILE_TRACK_FILE, 'w') as jfp:
            json.dump(self.created_files, jfp)

    def run(self, terms, variables=None, **kwargs):

        '''
        terms contains the content to be written to the temporary file
        '''
        try:
            self._load_created()

            ret = []
            for content in terms:
                content_sig = _hash_content(content)
                file_exists = False

                # Check if file was already create for this content and check it.
                if content_sig in self.created_files:
                    if os.path.exists(self.created_files[content_sig]):
                        with open(self.created_files[content_sig], 'r') as efh:
                            if content_sig == _hash_content(efh.read()):
                                file_exists = True
                                ret.append(self.created_files[content_sig])
                            else:
                                os.remove(self.created_files[content_sig])

                # Create / Replace the file
                if not file_exists:
                    temp_handle, temp_path = tempfile.mkstemp(text=True)
                    with os.fdopen(temp_handle, 'a') as temp_file:
                        temp_file.write(content)
                        self.created_files[content_sig] = temp_path
                        ret.append(temp_path)

            self._store_created()

            return ret

        except Exception as e:
            raise AnsibleError(to_native(repr(e)))

plugins/callback/clean_var_as_file.py

from ansible.plugins.callback import CallbackBase
from ansible_collections.thoteam.var_as_file.plugins.module_utils.var_as_file import VAR_AS_FILE_TRACK_FILE
from ansible.module_utils.common.text.converters import to_native
from ansible.errors import AnsibleError
import os
import json

def _make_clean():
    """Clean all files listed in VAR_AS_FILE_TRACK_FILE"""
    try:
        with open(VAR_AS_FILE_TRACK_FILE, 'r') as jfp:
            files = json.load(jfp)
            for f in files.values():
                os.remove(f)
        os.remove(VAR_AS_FILE_TRACK_FILE)
    except Exception as e:
        raise AnsibleError(to_native(repr(e)))

class CallbackModule(CallbackBase):
    ''' This Ansible callback plugin cleans-up files created by the thoteam.var_as_file.var_as_file lookup '''
    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = 'utility'
    CALLBACK_NAME = 'thoteam.var_as_file.clean_var_as_file'

    CALLBACK_NEEDS_WHITELIST = False
    # This one doesn't work for a collection plugin
    # Needs to be enabled anyway in ansible.cfg callbacks_enabled option
    CALLBACK_NEEDS_ENABLED = False

    def v2_playbook_on_stats(self, stats):
        _make_clean()

I'll be happy to get any feedback if your give it a try.