Azure DevOps Agent - Custom Setup/Teardown Operations

user_80630 0
Viewed 274

We have a cloud full of self-hosted Azure Agents running on custom AMIs. In some cases, I have some cleanup operations which I'd really like to do either before or after a job runs on the machine, but I don't want the developer waiting for the job to wait either at the beginning or the end of the job (which holds up other stages).

What I'd really like is to have the Azure Agent itself say "after this job finishes, I will run a set of custom scripts that will prepare for the next job, and I won't accept more work until that set of scripts is done".

In a pinch, maybe just a "cooldown" setting would work -- wait 30 seconds before accepting another job. (Then at least a job could trigger some background work before finishing.)

Has anyone had experience with this, or knows of a workable solution?

azure-devops-self-hosted-agent azure-devops
3 Answers

I suggest three solutions

  1. Create another pipeline to run the clean up tasks on agents - you can also add demand for specific agents (See here https://docs.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml) by Agent.Name -equals [Your Agent Name]. You can set frequency to minutes, hours, as you like using cron pattern. As while this pipeline will be running and taking up the agent, the agent being cleaned will not be available for other jobs. Do note that you can trigger this pipeline from another pipeline, but if both are using the same agents - they can just get deadlocked.

  2. Create a template containing scripts tasks having all clean up logic and use it at the end of every job (which you have discounted).

  3. Rather than using static VM's for agent hosting, use Azure scaleset for Self hosted agents - everytime agents are scaled down they are gone and when scaled up they start fresh. This saves a lot of money in terms of sitting agents not doing anything when no one is working. We use this option and went away from static agents. We have also used packer to create the VM image/vhd overnight to update it with patches, softwares required, and docker images cached. ref: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops

user_578640

For those discovering this question, there's a much better way: run your self-hosted agent with the --once flag, documented here.

You'll need to wrap it in your own bash script, but something like this works:

while :
do
  echo "Performing pre-job setup..."
  
  echo "Waiting for job..."
  ./run.sh --once

  echo "Cleaning up..."

  sleep 2
done
user_806300
user_7360790
Related