Ansible is getting stuck on some server which is in bad ps state?

user_9049216 0
Viewed 773

I have an ansible playbook as shown below and it works fine most of the times. But recently what I am noticing is it is getting stuck on some of the servers from the ALL group and just sits there. It doesn't even move forward to other servers in the ALL list.

# This will copy files
---
- hosts: ALL
  serial: "{{ num_serial }}"
  tasks:
      - name: copy files
        shell: "(ssh -o StrictHostKeyChecking=no abc.com 'ls -1 /var/lib/jenkins/workspace/copy/stuff/*' | parallel -j20 'scp -o StrictHostKeyChecking=no abc.com:{} /data/records/')"

      - name: sleep for 5 sec
        pause: seconds=5

So when I started debugging, I noticed on the actual server it is getting stuck - I can ssh (login) fine but when I run ps command then it just hangs and I don't get my cursor back so that means ansible is also getting stuck executing above scp command on that server.

So my question is even if I have some server in that state, why not just Ansible times out and move to other server? IS there anything we can do here so that ansible doesn't pause everything just waiting for that server to respond.

Note server is up and running and I can ssh fine but when we run ps command it just hangs and because of that Ansible is also hanging.

Is there any way to run this command ps aux | grep app on all the servers in ALL group and make a list of all the servers which executed this command fine (and if gets hang on some server then time out and move to other server in ALL list) and then pass on that list to work with my above ansible playbook? Can we do all this in one playbook?

ansible
3 Answers

Ansible doesn't have this feature and it might even be dangerous to have it. My suggestion in this case would be: see the failure, rebuild the server, run again.

It's possible to to build the feature you want in your playbook, what you could do is to have a dummy async task that triggers the issue, and the verify the outcome of that. If the async task didn't finish in a reasonable time, use the meta: end_host task to move to the next host.

You might need to mark some of those tasks with ignore_errors: yes.

Sorry that I cannot give you a complete answer as I've never tried to do this.

user_6088200

You can use strategies to achieve your goal. By default:

Plays run with a linear strategy, in which all hosts will run each task before any host starts the next task

By using the free strategy, each host will run until the end of the play as fast as it can. For example:

---
- hosts: ALL
  strategy: free
  tasks:
      - name: copy files
        shell: "(ssh -o StrictHostKeyChecking=no abc.com 'ls -1 /var/lib/jenkins/workspace/copy/stuff/*' | parallel -j20 'scp -o StrictHostKeyChecking=no abc.com:{} /data/records/')"

      - name: sleep for 5 sec
        pause: seconds=5

Another option would be to use timeout to run your command, then using registers to process whether the command executed successfully or not. For example timeout 5 sleep 10 returns 124 because of the timeout while timeout 5 sleep 3 returns 0 because the command terminates before the timeout occurs. In an ansible script, you could use something like:

  tasks:
     - shell: timeout 5 ps aux | grep app
       register: result
       ignore_errors: True

     - debug:
         msg: timeout occured
       when: result.rc == 124
user_113323700

As told by"Alassane Ndiaye ", you can try below code snippet. Where I am giving condition when shell is not timeout 

tasks:
     - shell: timeout 5 ps aux | grep app
       register: result
       ignore_errors: True

     - name: Run your shell command
       shell:  "(ssh -o StrictHostKeyChecking=no abc.com 'ls -1 /var/lib/jenkins/workspace/copy/stuff/*' | parallel -j20 'scp -o StrictHostKeyChecking=no abc.com:{} /data/records/')"
       when: result.rc != 124 && result.rc != 0
user_43527380
Related