How to compare two tarball's content

user_81754 0
Viewed 39451

I want to tell whether two tarball files contain identical files, in terms of file name and file content, not including meta-data like date, user, group.

However, There are some restrictions: first, I have no control of whether the meta-data is included when making the tar file, actually, the tar file always contains meta-data, so directly diff the two tar files doesn't work. Second, since some tar files are so large that I cannot afford to untar them in to a temp directory and diff the contained files one by one. (I know if I can untar file1.tar into file1/, I can compare them by invoking 'tar -dvf file2.tar' in file/. But usually I cannot afford untar even one of them)

Any idea how I can compare the two tar files? It would be better if it can be accomplished within SHELL scripts. Alternatively, is there any way to get each sub-file's checksum without actually untar a tarball?

Thanks,

compare compression tar linux
12 Answers

Are you controlling the creation of these tar files?
If so, the best trick would be to create a MD5 checksum and store it in a file within the archive itself. Then, when you want to compare two files, you just extract this checksum files and compare them.

If you can afford to extract just one tar file, you can use the --diff option of tar to look for differences with the contents of other tar file.

One more crude trick if you are fine with just a comparison of the filenames and their sizes.
Remember, this does not guarantee that the other files are same!

execute a tar tvf to list the contents of each file and store the outputs in two different files. then, slice out everything besides the filename and size columns. Preferably sort the two files too. Then, just do a file diff between the two lists.

Just remember that this last scheme does not really do checksum.

Sample tar and output (all files are zero size in this example).

$ tar tvfj pack1.tar.bz2
drwxr-xr-x user/group 0 2009-06-23 10:29:51 dir1/
-rw-r--r-- user/group 0 2009-06-23 10:29:50 dir1/file1
-rw-r--r-- user/group 0 2009-06-23 10:29:51 dir1/file2
drwxr-xr-x user/group 0 2009-06-23 10:29:59 dir2/
-rw-r--r-- user/group 0 2009-06-23 10:29:57 dir2/file1
-rw-r--r-- user/group 0 2009-06-23 10:29:59 dir2/file3
drwxr-xr-x user/group 0 2009-06-23 10:29:45 dir3/

Command to generate sorted name/size list

$ tar tvfj pack1.tar.bz2 | awk '{printf "%10s %s\n",$3,$6}' | sort -k 2
0 dir1/
0 dir1/file1
0 dir1/file2
0 dir2/
0 dir2/file1
0 dir2/file3
0 dir3/

You can take two such sorted lists and diff them.
You can also use the date and time columns if that works for you.

user_704820

tarsum is almost what you need. Take its output, run it through sort to get the ordering identical on each, and then compare the two with diff. That should get you a basic implementation going, and it would be easily enough to pull those steps into the main program by modifying the Python code to do the whole job.

user_1182400

There is also diffoscope, which is more generic, and allows to compare things recursively (including various formats).

pip install diffoscope
user_9868960

Is tardiff what you're looking for? It's "a simple perl script" that "compares the contents of two tarballs and reports on any differences found between them."

user_98780

I propose gtarsum, that I have written in Go, which means it will be an autonomous executable (no Python or other execution environment needed).

go get github.com/VonC/gtarsum

It will read a tar file, and:

  • sort the list of files alphabetically,
  • compute a SHA256 for each file content,
  • concatenate those hashes into one giant string
  • compute the SHA256 of that string

The result is a "global hash" for a tar file, based on the list of files and their content.

It can compare multiple tar files, and return 0 if they are identical, 1 if they are not.

VonC_63090

Just throwing this out there since none of the above solutions worked for what I needed.

This function gets the md5 hash of the md5 hashes of all the file-paths matching a given path. If the hashes are the same, the file hierarchy and file lists are the same.

I know it's not as performant as others, but it provides the certainty I needed.

PATH_TO_CHECK="some/path"
for template in $(find build/ -name '*.tar'); do
    tar -xvf $template --to-command=md5sum | 
        grep $PATH_TO_CHECK -A 1 | 
        grep -v $PATH_TO_CHECK | 
        awk '{print $1}' | 
        md5sum | 
        awk "{print \"$template\",\$1}"
done

*note: An invalid path simply returns nothing.

user_5927600
Related