Synchronize corresponding directories in two repositories (to achieve access control)

Question

Extracted Data from Code

My git repository contains both code and resources. The project is Scala, but Java would be similar: the resources are contained in shared/src/main/resources path, the rest are sources. Let us call this repository "code repository".

I want some people to be able to edit the resources, but not to have any access to the rest of the repository. To achieve this I have created a new repository (let us call it "data") and committed two commits into it:

an initial commit from the code repository (I was lucky this commit did not contain any files at all), therefore including it was easy (this prevents unrelated history error)
a commit with the current state of shared/src/main/resources

People who contribute only to data get access only to the "data" repository.

Merging Data to Code

This works quite fine, I am able to merge any changes done in the "data" repository to my "code" repository without any issues.

Merging data changes from Code to Data

The only trouble is when I make any changes to resources in my "code" repository. When I attempt to merge such changes, git wants to merge all commits which I previously did not include in the "data" repository, effectively adding all the code to it. I can fix such merge by using --no-ff and --no-commit options and reverting the unwanted file insertions before committing the merge. The result obtained this way looks good in the "data" repository, but then merging from "data" to "code" become difficult, as this "reverting" is now part of the merge commit and git wants to apply it. I can prevent it the same way, but this again only brings the problem to the other side.

Is there any workflow or tool which could be used for such situation? I am aware of submodules, but I would like to stay clear of them, as they bring their own problems.

I am aware this is probably not intended use of Git and not something which is normally done and I am kind expecting the answer "no, this is not possible", but seeing the sheer amount of creativity some people show when using Git I told myself I will ask here with a hope some solution can be found.

user_2591803 · Answer

I can see two main ways of doing this without using submodules, depending on whether you want to keep the history or not. Both rely on the same idea: create a new branch with only changes to the files you're interested in, then merge that branch. I'm going to assume both repositories contain a shared/src/main/resources path with the relevant folders: if you moved the one in "data" to the root it gets more complex because the commits relate to files in different locations. One easy-ish fix is to create a commit in "data" moving the files to shared/src/main/resources, merge in changes from "code", then remove the commit (using git rebase -i). Alternatively in "keeping history" you can use git-filter-repo to rewrite the folder location.

Losing history

This is easier to do as a one-off, but harder to automate, and may lead to harder merges. Here you do the same thing you did when creating the "data" repository: just dump the existing state of the folder over the top of the data repository. If you've just merged "data" into "code", then "code" contains the changes you want: just copy the resources file into "data" and create a commit.

Keeping history

Make another repo / branch containing only the changes to shared/src/main/resources. I recommend using git-filter-repo. Note that this will rewrite your repository: I'm not sure if you can get it to only rewrite a branch.

cd /path/to/new/repo
git clone code
git checkout -b resources-only
git filter-repo --path shared/src/main/resources

Then, in "data":

git remote add code-resources-only /path/to/new/repo/code
git fetch code-resources-only
git merge --allow-unrelated-histories code-resources-only/resources-only
git remote remove code-resources-only

One minor problem with the "data" repository is that it contains a commit not in the history of "code":

a commit with the current state of shared/src/main/resources

while "code" would have many commits building up the resources folder. This isn't really a problem after the first merge, but might be nicer if the branches really shared history.