Detach (move) subdirectory into separate Git repository

Update: This process is so common, that the git team made it much simpler with a new tool, git subtree. See here: Detach (move) subdirectory into separate Git repository

You want to clone your repository and then use git filter-branch to mark everything but the subdirectory you want in your new repo to be garbage-collected.

1. To clone your local repository:

   git clone /XYZ /ABC
   

   (Note: the repository will be cloned using hard-links, but that is not a problem since the hard-linked files will not be modified in themselves - new ones will be created.)

2. Now, let us preserve the interesting branches which we want to rewrite as well, and then remove the origin to avoid pushing there and to make sure that old commits will not be referenced by the origin:

   cd /ABC
   for i in branch1 br2 br3; do git branch -t $i origin/$i; done
   git remote rm origin
   

   or for all remote branches:

   cd /ABC
   for i in $(git branch -r | sed "s/.*origin\///"); do git branch -t $i origin/$i; done
   git remote rm origin
   

3. Now you might want to also remove tags which have no relation with the subproject; you can also do that later, but you might need to prune your repo again. I did not do so and got a WARNING: Ref 'refs/tags/v0.1' is unchanged for all tags (since they were all unrelated to the subproject); additionally, after removing such tags more space will be reclaimed. Apparently git filter-branch should be able to rewrite other tags, but I could not verify this. If you want to remove all tags, use git tag -l | xargs git tag -d.

4. Then use filter-branch and reset to exclude the other files, so they can be pruned. Let's also add --tag-name-filter cat --prune-empty to remove empty commits and to rewrite tags (note that this will have to strip their signature):

   git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC -- --all
   

   or alternatively, to only rewrite the HEAD branch and ignore tags and other branches:

   git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC HEAD
   

5. Then delete the backup reflogs so the space can be truly reclaimed (although now the operation is destructive)

   git reset --hard
   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
   git reflog expire --expire=now --all
   git gc --aggressive --prune=now
   

   and now you have a local git repository of the ABC sub-directory with all its history preserved.

Note: For most uses, git filter-branch should indeed have the added parameter -- --all. Yes that's really --space-- all. This needs to be the last parameters for the command. As Matli discovered, this keeps the project branches and tags included in the new repo.

Edit: various suggestions from comments below were incorporated to make sure, for instance, that the repository is actually shrunk (which was not always the case before).

Paul's answer creates a new repository containing /ABC, but does not remove /ABC from within /XYZ. The following command will remove /ABC from within /XYZ:

git filter-branch --tree-filter "rm -rf ABC" --prune-empty HEAD

Of course, test it in a 'clone --no-hardlinks' repository first, and follow it with the reset, gc and prune commands Paul lists.

When running git filter-branch using a newer version of git (2.22+ maybe?), it says to use this new tool git-filter-repo. This tool certainly simplified things for me.

Filtering with filter-repo

Commands to create the XYZ repo from the original question:

# create local clone of original repo in directory XYZ
tmp $ git clone git@github.com:user/original.git XYZ

# switch to working in XYZ
tmp $ cd XYZ

# keep subdirectories XY1 and XY2 (dropping ABC)
XYZ $ git filter-repo --path XY1 --path XY2

# note: original remote origin was dropped
# (protecting against accidental pushes overwriting original repo data)

# XYZ $ ls -1
# XY1
# XY2

# XYZ $ git log --oneline
# last commit modifying ./XY1 or ./XY2
# first commit modifying ./XY1 or ./XY2

# point at new hosted, dedicated repo
XYZ $ git remote add origin git@github.com:user/XYZ.git

# push (and track) remote master
XYZ $ git push -u origin master

assumptions: * remote XYZ repo was new and empty before the push

Filtering and moving

In my case, I also wanted to move a couple of directories for a more consistent structure. Initially, I ran that simple filter-repo command followed by git mv dir-to-rename, but I found I could get a slightly "better" history using the --path-rename option. Instead of seeing last modified 5 hours ago on moved files in the new repo I now see last year (in the GitHub UI), which matches the modified times in the original repo.

Instead of...

git filter-repo --path XY1 --path XY2 --path inconsistent
git mv inconsistent XY3  # which updates last modification time

I ultimately ran...

git filter-repo --path XY1 --path XY2 --path inconsistent --path-rename inconsistent:XY3

• I thought the Git Rev News blog post explained well the reasoning behind creating yet another repo-filtering tool.
• I initially tried the path of creating a sub-directory matching the target repo name in the original repository and then filtering (using git filter-repo --subdirectory-filter dir-matching-new-repo-name). That command correctly converted that subdirectory to the root of the copied local repo, but it also resulted in a history of only the three commits it took to create the subdirectory. (I hadn't realized that --path could be specified multiple times; thereby, obviating the need to create a subdirectory in the source repo.) Since someone had committed to the source repo by the time I noticed that I'd failed to carry forward the history, I just used git reset commit-before-subdir-move --hard after the clone command, and added --force to the filter-repo command to get it to operate on the slightly modified local clone.

git clone ...
git reset HEAD~7 --hard      # roll back before mistake
git filter-repo ... --force  # tell filter-repo the alterations are expected

• I was stumped on the install since I was unaware of the extension pattern with git, but ultimately I cloned git-filter-repo and symlinked it to $(git --exec-path):

ln -s ~/github/newren/git-filter-repo/git-filter-repo $(git --exec-path)

Update: The git-subtree module was so useful that the git team pulled it into core and made it git subtree. See here: Detach (move) subdirectory into separate Git repository

git-subtree may be useful for this

http://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt (deprecated)

http://psionides.jogger.pl/2010/02/04/sharing-code-between-projects-with-git-subtree/

To add to Paul's answer, I found that to ultimately recover space, I have to push HEAD to a clean repository and that trims down the size of the .git/objects/pack directory.

i.e.

$ mkdir ...ABC.git
$ cd ...ABC.git
$ git init --bare

After the gc prune, also do:

$ git push ...ABC.git HEAD

Then you can do

$ git clone ...ABC.git

and the size of ABC/.git is reduced

Actually, some of the time consuming steps (e.g. git gc) aren't needed with the push to clean repository, i.e.:

$ git clone --no-hardlinks /XYZ /ABC
$ git filter-branch --subdirectory-filter ABC HEAD
$ git reset --hard
$ git push ...ABC.git HEAD

You might need something like "git reflog expire --expire=now --all" before the garbage collection to actually clean the files out. git filter-branch just removes references in the history, but doesn't remove the reflog entries that hold the data. Of course, test this first.

My disk usage dropped dramatically in doing this, though my initial conditions were somewhat different. Perhaps --subdirectory-filter negates this need, but I doubt it.

You can easily try the https://help.github.com/enterprise/2.15/user/articles/splitting-a-subfolder-out-into-a-new-repository/

This worked for me. The issues i faced in the steps given above are

1. in this command git filter-branch --prune-empty --subdirectory-filter FOLDER-NAME BRANCH-NAME The BRANCH-NAME is master

2. if the last step fails when committing due to protection issue follow - https://docs.gitlab.com/ee/user/project/protected_branches.html

I've found quite straight forward solution, The idea is to copy repository and then just remove unnecessary part. This is how it works:

1) Clone a repository you'd like to split

git clone git@git.thehost.io:testrepo/test.git

2) Move to git folder

cd test/

2) Remove unnecessary folders and commit it

rm -r ABC/
git add .
enter code here
git commit -m 'Remove ABC'

3) Remove unnecessary folder(s) form history with BFG

cd ..
java -jar bfg.jar --delete-folders "{ABC}" test
cd test/
git reflog expire --expire=now --all && git gc --prune=now --aggressive

for multiply folders you can use comma

java -jar bfg.jar --delete-folders "{ABC1,ABC2}" metric.git

4) Check that history doesn't contains the files/folders you just deleted

git log --diff-filter=D --summary | grep delete

5) Now you have clean repository without ABC, so just push it into new origin

remote add origin git@github.com:username/new_repo
git push -u origin master

That's it. You can repeat the steps to get another repository,

just remove XY1,XY2 and rename XYZ -> ABC on step 3

Found this wonderful article Original reference easy to follow. Documenting it here in case if it get's inaccessible.

1. Preparing the current repository

$ cd path/to/repository
$ git subtree split -P my-folder -b my-folder
Created branch 'my-folder'
aecbdc3c8fe2932529658f5ed40d95c135352eff

The name of the folder must be a relative path, starting from the root of the repository.

2. Creating the new repository

$ cd my-folder
$ git init
Initialized empty Git repository in /Users/adamwest/Projects/learngit/shop/my-folder/.git/
$ git add .
$ git commit -m "initial commit"
[master (root-commit) 192c10b] initial commit
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 file

Here we just need to cd to the new folder, initialise the new repository, and commit any contents.

3.Add new remote repository and push

$ git remote add origin git@github.com:robertlyall/my-folder.git
$ git push origin -u master
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 199 bytes | 199.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To github.com:robertlyall/my-folder.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

We add the new repository remote from GitHub here, then push our first commit to it.

4. Remove folder from main repository and push

$ cd ../
$ git rm -rf my-folder
rm 'my-folder/file'
$ git commit -m "Remove old folder"
[master 56aedbe] remove old folder
 1 file changed, 0 insertions(+), 0 deletions(-)
 delete mode 100644 my-folder/file
$ git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 217 bytes | 217.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To github.com:robertlyall/shop.git
   74dd8b3..56aedbe  master -> master

Finally, we cd back to the rooot directory, remove the folder from our main repository, then commit and push the change. Now, we have the folder in our main repository but linked to a completely separate repository that can be reused across multiple projects.