Large size of the tar files generated by python tarfile

yaofo chen 0
Viewed 19

Aftering generating some tar files using webdataset, I select one tar file(webdataset shard) named 000000.tar (18MB) to unpack and then repack into a new tarfile example.tar (15MB) in the shell (see commands in the followings). The repakced tarfile example.tar is smaller than the original tar file 000000.tar by 3MB. I think these two tarfiles include totally the same things. And I have no idea why 000000.tar is larger than example.tar.

This problem results in that tarfiles generated by webdataset occupy more disk space, which is inefficient. In fact, webdataset uses python built-in tarfile library to generate tarfile. There must be something different between python built-in tarfile library and the tar command in the shell. But I don't know how to figure it out.

  • Supplementary:

I put these two files (000000.tar and example.tar) in Google Drive. And I provide the command I used to generate example.tar in the followings.

(base) {22-09-01 2:25}vm:~/playground chenyaofo% ls -lha
total 18M
drwxrwxr-x  2 chenyaofo chenyaofo 4.0K Sep  1 02:25 .
drwxr-xr-x 39 chenyaofo chenyaofo 4.0K Sep  1 02:25 ..
-rw-rw-r--  1 chenyaofo chenyaofo  18M Sep  1 02:19 000000.tar
(base) {22-09-01 2:25}vm:~/playground chenyaofo% mkdir images
(base) {22-09-01 2:25}vm:~/playground chenyaofo% tar -xf 000000.tar -C images
(base) {22-09-01 2:26}vm:~/playground chenyaofo% tar -cf example.tar -C images .
(base) {22-09-01 2:26}vm:~/playground chenyaofo% ls -lha
total 33M
drwxrwxr-x  3 chenyaofo chenyaofo 4.0K Sep  1 02:26 .
drwxr-xr-x 39 chenyaofo chenyaofo 4.0K Sep  1 02:26 ..
-rw-rw-r--  1 chenyaofo chenyaofo  18M Sep  1 02:19 000000.tar
-rw-rw-r--  1 chenyaofo chenyaofo  15M Sep  1 02:26 example.tar
drwxrwxr-x  2 chenyaofo chenyaofo 160K Sep  1 02:26 images

In addition, I provide some system info and library info. I use Ubuntu 22.04 Linux vm 5.15.0-39-generic #42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux. I use python=3.9.13 and webdataset=0.2.20 to generate webdataset shards. The version of tar command is tar (GNU tar) 1.34.

The code used to generate webdataset shards is available in Github Gist.

webdataset shell tar linux
1 Answers

I maybe found the reason for the large tarfile size.

I found a similar question in Stackoverflow. In short, python (>=3.8) built-in library uses tarfile.PAX_FORMAT as default to store tarfile. In constrast, the default format of tar command on Linux is tarfile.GNU_FORMAT. See more about these two formats here and here. PAX_FORMAT has a longer tar header than GNU_FORMAT and thus lead to large tarfile size.

yaofo chen0
Related