Get consistent md5 checksum of gzipped file using data.table and R.utils

user_13259574 0
Viewed 186

I'm trying to replicate the checksum of a compressed file with R.utils::gzip and the compress = "gzip" argument of data.table::fwrite but I keep getting different results. Here is an example

library(data.table); library(R.utils); library(digest)

dt <- data.frame(a = c(1, 2, 3))

fwrite(dt, "r-utils.csv")
gzip("r-utils.csv")
fwrite(dt, "datatable-v1.csv.gz", compress = "gzip")

digest(file = "r-utils.csv.gz")
#> [1] "8d4073f4966f94ac5689c6e85de2d92d"
digest(file = "datatable-v1.csv.gz")
#> [1] "5d58f9eeefb166c6d50ac00f3215e689"

Initially I though that fwrite was storing the filename and timestamp in the output file (per the usual gzip behaviour without the --no-name option), but that don't appear to be the case since I get the same checksum in different calls to fwrite

fwrite(dt, "datatable-v2.csv.gz", compress = "gzip")
digest(file = "datatable-v2.csv.gz")
#> [1] "5d58f9eeefb166c6d50ac00f3215e689"

Any ideas on what might be causing the difference?

PS. Incidentally, the checksum of the uncompressed files are the same

$ md5sum datatable-v1.csv 
7e034138dc91aa575d929c1ed65aa67c  datatable-v1.csv
$ md5sum r-utils.csv 
7e034138dc91aa575d929c1ed65aa67c  r-utils.csv
data.table r gzip
1 Answers

If you look at the bytes, you can see they are different

r-utils.csv.gz
  Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F   
00000000: 1F 8B 08 00 00 00 00 00 00 06 4B E4 E5 32 E4 E5    ..........Kde2de
00000010: 32 E2 E5 32 E6 E5 02 00 21 EB 62 BF 0C 00 00 00    2be2fe..!kb?....

and

datatable-v2.csv.gz
  Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F   
00000000: 1F 8B 08 00 00 00 00 00 00 0A 4B E4 E5 02 00 56    ..........Kde..V
00000010: EF 2F E3 03 00 00 00 1F 8B 08 00 00 00 00 00 00    o/c.............
00000020: 0A 33 E4 E5 32 E2 E5 32 E6 E5 02 00 49 C4 FF 4D    .3de2be2fe..ID.M
00000030: 09 00 00 00                                        ....

So the data.table one is longer. This appears to be because the default compression settings are different. Specifically it looks like the two method use a different "window size" parameter. The data.table code uses a windowBits value of 31 (15+16) which will include "trailing checksum in the output" but the R.utils::gzip function uses the base R gzfile() function which uses a windowBits value of -15 (MAX_WBITS) and that negative value means a trailing checksum should not be used. So I think that accounts for the extra bytes in the in the data.table output.

Because you can use different compression level settings and checksums and gzip headers, it's not necessarily the case the you will get the same checksum for compressed versions of data files if two different compression pipelines are used. So it's possible for the data inside to be identical but the actual compressed files to be different.

Since these settings are part of the C code for the package and for base R this is not something you will be able to change in R code. It's not possible for these two different methods to return identical output.

user_23720640
Related