How to run AWS CLI commands inside jenkinsfile in a loop

Question

I am trying to create a jenkins job that when triggered will update all the Elasticache Redis replication groups with a certain tag.

The main workflow is that I find all the Redis replica groups in a region for example us-east-1

def findAllRedisReplicationGroups(label, region) {
  query = "ReplicationGroups[*].{arn: ARN}"

  output = sh(
    script: "aws --region ${region} elasticache describe-replication-groups --query '${query}' --output text",
    label: label,
    returnStdout: true
  )
  return output
}

The output will be a string example here

String a = """
          arn:aws:elasticache:us-west-2:AccountID:replicationgroup:application-2-test
          arn:aws:elasticache:us-west-2:AccountID:replicationgroup:application-1-test
          """

Then I split the string into a list with each arn being an element. Then using the for loop I will iterate through all the Redis replica groups and get their tags, if the tag is like Environment: test then the arn of the Redis replica group will be added to the list of arn

def findCorrectEnvReplicationGroups(label, region, environment, redis_arns){
  def arn_list = redis_arns.split();
  def correct_env_arn_list = [];
  for( def arn : arn_list) {
    def redisTags = getRedisTags(label, region, arn)

    def jsonSlurper = new groovy.json.JsonSlurper()
    def object = jsonSlurper.parseText(redisTags)

    EnvironmentFromTag = object.TagList.find { it.Key == "Environment"}

    if (EnvironmentFromTag.Value == environment) {
      correct_env_arn_list.add(arn)
    }
    break
  }
  return correct_env_arn_list
}

def getRedisTags(label, region, arn) {
  output = sh(
    script: "aws --region ${region} elasticache list-tags-for-resource --resource-name ${arn} --output json",
    label: label,
    returnStdout: true
  )

  return output
}

I get through 1 loop. Tested by printing out the arn for each cycle but it crashes when trying to run the script in the getRedisTags method again.

The output should be a list of arns whose tags matches

Has anyone come across such an error or has any experience with groovy and can maybe help me out on why the jenkinsfile crashes when trying to run the aws cli command in a loop

many thanks

user_15247774 · Answer

Still not totally sure why it didnt work but with using groovy-s built in iterators I got it working

First I get a list of all the Redis Replication Groups
Then insert that into findEnvironmentReplicationGroups where I use the .findAll to input the arns one by one into the method getRedisTags which returns me a map (example = [Environment: "test"]) each time
Then I store the output [Environment: "test"] into the variable tags and check if it matches the environment given to the method
If it matches then it is added to correctArns which is ultimately returned back

def findAllRedisReplicationGroups(region) {
  def query = "ReplicationGroups[*].ARN"

  def output = sh(
    script: "aws --region ${region} elasticache describe-replication-groups --query '${query}' --output json",
    label: "Find all Redis Replication groups in the region",
    returnStdout: true
  )
  readJSON(text: output)
}

def findEnvironmentReplicationGroups(region, environment, listOfRedisArns){
  def correctArns = listOfRedisArns.findAll { arn ->
    def tags = getRedisTags(region, arn).collectEntries { [it.Key, it.Value] }
    tags.Environment == environment
  }
  correctArns
}

def getRedisTags(region, arn) {
  def query = "TagList[?Key==`Environment`]"
  def output = sh(
    script: "aws --region ${region} elasticache list-tags-for-resource --resource-name ${arn} --query '${query}' --output json",
    label: "Get tags for a Redis replication group",
    returnStdout: true
  )
  readJSON(text: output)
}

How to run AWS CLI commands inside jenkinsfile in a loop

1 Answers