git finding duplicate commits (by patch-id)

user_1462529 0
Viewed 6611

I'd like a recipe for finding duplicated changes. patch-id is likely to be the same but the commit attributes may not be.

This seems to be an intended use of patch-id:

git patch-id --help

IOW, you can use this thing to look for likely duplicate commits.

I imagine that stringing together "git log", "git patch-id" and uniq could do the job badly but if someone has an command that does the job well, I'd appreciate it.

duplicates commit git
7 Answers

For anyone wanting to do this on windows powershell the equivalent command to unagi's answer is:

git rev-list --no-merges --all  | %{&git.exe show $_} | 
  git patch-id | ConvertFrom-String -PropertyNames PatchId, Commit | 
  Group-Object PatchId | Where-Object count -gt 1 | 
  %{$_.group.Commit + " "}

Gives an output like:

1605e0e1e13d7b3f456c20432d8edec664ca7117
1e8efa8f2f01962a2c08fd25caf687d330383428

b45b6db084b27ae420ac8e9cf6511110ebb46513
4a2e1e3ba5a9a1d5db1d00343813e1404f6124e2

With the duplicate commit hashes grouped together.

CAUTION: On my repo this was a slow command so be sure to filter the call to rev-list appropriately!

user_4701830

Make sure to use a recent version of Git

The git log --format=%H mentioned by the OP bsb's answer is not always unique.

That is because, before Git 2.29 (Q4 2020), the patch-id computation did not ignore the "incomplete last line" marker like whitespaces.

See commit 82a6201 (19 Aug 2020) by René Scharfe (rscharfe).
(Merged by Junio C Hamano -- gitster -- in commit 5122614, 24 Aug 2020)

patch-id: ignore newline at end of file in diff_flush_patch_id()

Reported-by: Tilman Vogel
Initial-test-by: Tilman Vogel
Signed-off-by: René Scharfe

Whitespace is ignored when calculating patch IDs.
This is done by removing all whitespace from diff lines before hashing them, including a newline at the end of a file.
If that newline is missing, however, diff reports that fact in a separate line containing "\ No newline at end of file\n", and this marker is hashed like a context line.

This goes against our goal of making patch IDs independent of whitespace.

Use the same heuristic that 2485eab55cc (git-patch-id: do not trip over "no newline" markers, 2011-02-17) added to git patch-id(man) instead and skip diff lines that start with a backslash and a space and are longer than twelve characters.

VonC_63090
Related