Occasional "fatal: not a git repository" in a git worktree

ElpieKay 0
Viewed 36

I have 4 Ubuntu machines and a Jenkins job runs on them. A shared disk is mounted to them as /data/repositories/. Under it, there are many non-bare repositories created by git clone <url> --no-checkout -b master.

When the job is triggered, it runs these step:

  1. In /data/repositories/foo, fetch Branch A and Branch B
  2. Parse Branch B head and get its commit SHA1VALUE
  3. Create a work tree by git worktree add --no-checkout /path/to/worktree SHA1VALUE
  4. Get the changed files in Branch A head
  5. Enable sparse checkout in /path/to/worktree
  6. Write the changed files into /data/repositories/foo/.git/worktrees/xx/info/sparse-checkout
  7. Run git checkout in /path/to/worktree to check out these files
  8. Use git cherry-pick to apply Branch A head onto SHA1VALUE in /path/to/worktree
  9. Push the new commit to Branch B.

/path/to/worktree is not on the shared disk but on each machine's own disk. In Step 7 and Step 9, it may occasionally raise an error fatal: not a git repository /data/repositories/foo/.git/worktrees/xx. When the error occurs, /data/repositories/foo/.git/worktrees/xx is not there as if it's been deleted by some process or thread right before the step. In the job, it won't be deleted on purpose until the job is finished or an exception is raised in these steps.

If I rebuild the job with the same parameters, the problem does not occur again.

If /data/repositories/foo/.git/worktrees/xx is not successfully created in the first place, it would be detected during these steps and Step 8 would always fail.

Multiple jobs could run on the same /data/repositories/foo at the same time. But in /data/repositories/foo/.git/worktrees/xx/info/sparse-checkout, the xx part is unique.

Thanks for any clues.

git-worktree git
1 Answers

Thanks to @torek. Although it's still not clear how exactly the problem is caused, I found a solution to fix it. According to the solution, the root cause is high likely related with race condition and file system types as @torek pointed out.

The shared disk is NFS and the non-shared is EXT4. So, the main repository resides in NFS system and its worktrees are created in EXT4 system. Under some unknown condition with concurrent jobs running, the worktree gitdir in the main repository gets deleted unexpectedly, and of course the worktree is disabled.

The solution is to create the worktrees in the shared disk, so that the main repository and worktrees are on the same disk. After testing for about 8 hours, the problem does not occur again.

ElpieKay0
Related