How to make shallow git submodules?

Following Ryan's answer I was able to come up with this simple script which iterates through all submodules and shallow clones them:

#!/bin/bash
git submodule init
for i in $(git submodule | sed -e 's/.* //'); do
    spath=$(git config -f .gitmodules --get submodule.$i.path)
    surl=$(git config -f .gitmodules --get submodule.$i.url)
    git clone --depth 1 $surl $spath
done
git submodule update

Reading through the git-submodule "source", it looks like git submodule add can handle submodules that already have their repositories present. In that case...

$ git clone $remote1 $repo
$ cd $repo
$ git clone --depth 5 $remotesub1 $sub1
$ git submodule add $remotesub1 $sub1
#repeat as necessary...

You'll want to make sure the required commit is in the submodule repo, so make sure you set an appropriate --depth.

Edit: You may be able to get away with multiple manual submodule clones followed by a single update:

$ git clone $remote1 $repo
$ cd $repo
$ git clone --depth 5 $remotesub1 $sub1
#repeat as necessary...
$ git submodule update

Summary of buggy / unexpected / annoying behaviour as of Git 2.14.1

1. shallow = true in .gitmodules only affects git clone --recurse-submodules if the HEAD of the remote submodule points to the required commit, even if the target commit is pointed to by a branch, and even if you put branch = mybranch on the .gitmodules as well.

   Local test script. Same behaviour on GitHub 2017-11, where HEAD is controlled by the default branch repo setting:

   git clone --recurse-submodules https://github.com/cirosantilli/test-shallow-submodule-top-branch-shallow
   cd test-shallow-submodule-top-branch-shallow/mod
   git log
   # Multiple commits, not shallow.
   

2. git clone --recurse-submodules --shallow-submodules fails if the commit is neither referenced by a branch or tag with a message: error: Server does not allow request for unadvertised object.

   Local test script. Same behaviour on GitHub:

   git clone --recurse-submodules --shallow-submodules https://github.com/cirosantilli/test-shallow-submodule-top-sha
   # error
   

   I also asked on the mailing list: https://marc.info/?l=git&m=151863590026582&w=2 and the reply was:

   In theory this should be easy. :)

   In practice not so much, unfortunately. This is because cloning will just obtain the latest tip of a branch (usually master). There is no mechanism in clone to specify the exact sha1 that is wanted.

   The wire protocol supports for asking exact sha1s, so that should be covered. (Caveat: it only works if the server operator enables uploadpack.allowReachableSHA1InWant which github has not AFAICT)

   git-fetch allows to fetch arbitrary sha1, so as a workaround you can run a fetch after the recursive clone by using "git submodule update" as that will use fetches after the initial clone.

TODO test: allowReachableSHA1InWant.

Are the canonical locations for your submodules remote? If so, are you OK with cloning them once? In other words, do you want the shallow clones just because you are suffering the wasted bandwidth of frequent submodule (re)clones?

If you want shallow clones to save local diskspace, then Ryan Graham's answer seems like a good way to go. Manually clone the repositories so that they are shallow. If you think it would be useful, adapt git submodule to support it. Send an email to the list asking about it (advice for implementing it, suggestions on the interface, etc.). In my opinion, the folks there are quite supportive of potential contributors that earnestly want to enhance Git in constructive ways.

If you are OK with doing one full clone of each submodule (plus later fetches to keep them up to date), you might try using the --reference option of git submodule update (it is in Git 1.6.4 and later) to refer to local object stores (e.g. make --mirror clones of the canonical submodule repositories, then use --reference in your submodules to point to these local clones). Just be sure to read about git clone --reference/git clone --shared before using --reference. The only likely problem with referencing mirrors would be if they ever end up fetching non-fast-forward updates (though you could enable reflogs and expand their expiration windows to help retain any abandoned commits that might cause a problem). You should not have any problems as long as

• you do not make any local submodule commits, or
• any commits that are left dangling by non-fast-forwards that the canonical repositories might publish are not ancestors to your local submodule commits, or
• you are diligent about keeping your local submodule commits rebased on top of whatever non-fast-forwards might be published in the canonical submodule repositories.

If you go with something like this and there is any chance that you might carry local submodule commits in your working trees, it would probably be a good idea to create an automated system that makes sure critical objects referenced by the checked-out submodules are not left dangling in the mirror repositories (and if any are found, copies them to the repositories that need them).

And, like the git clone manpage says, do not use --reference if you do not understand these implications.

# Full clone (mirror), done once.
git clone --mirror $sub1_url $path_to_mirrors/$sub1_name.git
git clone --mirror $sub2_url $path_to_mirrors/$sub2_name.git

# Reference the full clones any time you initialize a submodule
git clone $super_url super
cd super
git submodule update --init --reference $path_to_mirrors/$sub1_name.git $sub1_path_in_super
git submodule update --init --reference $path_to_mirrors/$sub2_name.git $sub2_path_in_super

# To avoid extra packs in each of the superprojects' submodules,
#   update the mirror clones before any pull/merge in super-projects.
for p in $path_to_mirrors/*.git; do GIT_DIR="$p" git fetch; done

cd super
git pull             # merges in new versions of submodules
git submodule update # update sub refs, checkout new versions,
                     #   but no download since they reference the updated mirrors

Alternatively, instead of --reference, you could use the mirror clones in combination with the default hardlinking functionality of git clone by using local mirrors as the source for your submodules. In new super-project clones, do git submodule init, edit the submodule URLs in .git/config to point to the local mirrors, then do git submodule update. You would need to reclone any existing checked-out submodules to get the hardlinks. You would save bandwidth by only downloading once into the mirrors, then fetching locally from those into your checked-out submodules. The hard linking would save disk space (although fetches would tend to accumulate and be duplicated across multiple instances of the checked-out submodules' object stores; you could periodically reclone the checked-out submodules from the mirrors to regain the disk space saving provided by hardlinking).

Shallow clone of a submodule is perfect because they snapshot at a particular revision/changeset. It's easy to download a zip from the website so I tried for a script.

#!/bin/bash
git submodule deinit --all -f
for value in $(git submodule | perl -pe 's/.*(\w{40})\s([^\s]+).*/\1:\2/'); do
  mysha=${value%:*}
  mysub=${value#*:}
  myurl=$(grep -A2 -Pi "path = $mysub" .gitmodules | grep -Pio '(?<=url =).*/[^.]+')
  mydir=$(dirname $mysub)
  wget $myurl/archive/$mysha.zip
  unzip $mysha.zip -d $mydir
  test -d $mysub && rm -rf $mysub
  mv $mydir/*-$mysha $mysub
  rm $mysha.zip
done
git submodule init

git submodule deinit --all -f clears the submodule tree which allows the script to be reusable.

git submodule retrieves the 40 char sha1 followed by a path that corresponds to the same in .gitmodules. I use perl to concatenate this information, delimited by a colon, then employ variable transformation to separate the values into mysha and mysub.

These are the critical keys because we need the sha1 to download and the path to correlate the url in .gitmodules.

Given a typical submodule entry:

[submodule "label"]
    path = localpath
    url = https://github.com/repository.git

myurl keys on path = then looks 2 lines after to get the value. This method may not work consistently and require refinement. The url grep strips any remaining .git type references by matching to the last / and anything up to a ..

mydir is mysub minus a final /name which would by the directory leading up to the submodule name.

Next is a wget with the format of downloadable zip archive url. This may change in future.

Unzip the file to mydir which would be the subdirectory specified in the submodule path. The resultant folder will be the last element of the url-sha1.

Check to see if the subdirectory specified in the submodule path exists and remove it to allow renaming of the extracted folder.

mv rename the extracted folder containing our sha1 to its correct submodule path.

Delete downloaded zip file.

Submodule init

This is more a WIP proof of concept rather than a solution. When it works, the result is a shallow clone of a submodule at a specified changeset.

Should the repository re-home a submodule to a different commit, re-run the script to update.

The only time a script like this would be useful is for non-collaborative local building of a source project.

I needed a solution to shallow clone submodules when I can not effect on cloning of main repo. Based on one solution above:

#!/bin/bash
git submodule init
for i in $(git submodule | sed -e 's/.* //'); do
    git submodule update --init --depth 1 -- $i
done