How to configure Kubernetes initContainers runs after the other finished

Loc Truong 0
Viewed 47

I have a situation where I want to speed up deployment time by caching the git resources into a shared PVC.

This is the bash script I use for checkout the resource and save into a share PVC folder

#!/bin/bash
src="$1"
dir="$2"

echo "Check for existence of directory: $dir"
if [ -d "$dir" ]
   then
     echo "$dir found, no need to clone the git"
  else
     echo "$dir not found, clone $src into $dir"
     mkdir -p $dir
     chmod -R 777 $dir
     git clone $src $dir
     echo "cloned $dir"
fi

Given I have a Deployment with more than 1 pods and each of them have an initContainer. The problem with this approach is all initContainers will start almost at the same time.

They all check for the existence of the git resource directory. Let's say first deployment we dont have the git directory yet. Then it will create the directory, then clone the resource. Now, the second and third initContainers see that the directory is already there so they finish immediately.

Is there a way to make other initContainers wait for the first one to finish? After reading the kubernetes documentation, I don't think it's supported by default

Edit 1:

  • The second solution I can think of is to deploy with 1 pod only, after a successful deployment, we will scale it out automatically. However I still don't know how to do this
kubernetes
1 Answers

I have found a workaround. The idea is to create a lock file, and write a script to wait until the lock file exists. In the initContainer, I prepare the script like this

#!/bin/bash
src="$1"
dir="$2"

echo "Check for existence of directory: $dir/src"
if [ -d "$dir/src" ]
   then
      echo "$dir/src found, check if .lock is exist"
      until [ ! -f $dir/.lock ]
      do
         sleep 5
         echo 'After 5 second, .lock is still there, I will check again'
      done
      echo "Finish clone in other init container, I can die now"
      exit
   else
      echo "$dir not found, clone $src into $dir"
      mkdir -p $dir/src
      echo "create .lock, make my friends wait for me"
      touch $dir/.lock
      ls -la $dir
      chmod -R 777 $dir
      git clone $src $dir/src
      echo "cloned $dir"
      echo "remove .lock now"
      rm $dir/.lock
fi

This kind of like a cheat, but it works. The script will make other initContainers wait until the .lock is remove. By then, the project is cloned already.

Loc Truong0
Related