Different result in GZIP decompress (PHP / Python)

user_5216168 0
Viewed 49

I'm having a gzipped file that seems to be missing characters when I uncompress the data in PHP. However when I do this for e.g. in Python the characters are present. In this case the last 2 characters (']}') are missing, which makes the data an invalid JSON.

Below 2 examples that both output different results; how is it possible that PHP doesn't output the full data? I've also inspected the GZIP content and it looks like the length and CRC are not valid; at least when I compare the structure with the data from this site: https://docs.fileformat.com/compression/gz/

<?php

$base64gzip = 'H4sIAAAAAAAAA7XSO04DQAwE0J5jbE3h8d9cJUoRiSPQcXmEFKR4KFKlfdr12Lv+Pp+3r9v5uFwORnTSACTO9f3tUcKfiLbOsFQsEVMxEkwtMRNROtO9b4V0yT5TAcoq5O5QBtkssSeV6LxnSUXWVEKDp5jmDocmNa3e6ZapYJE9OypCWXxXDo0BC2VJKr+qT3AdTXsialw5BHb/5ZBwbRNHWm2C51/aA3n/IwNX9+RJaveE369mcdrOmNFXScdLxKUg64GgkkqLFrz4JjJ+rj/Cx4XMxgMAAB+LCAAAAAAAAAOLrQUAZzx4YAIAAAA=';

echo gzdecode(base64_decode($base64gzip));

#!/usr/bin/python

import gzip
import base64

base64gzip = '''H4sIAAAAAAAAA7XSO04DQAwE0J5jbE3h8d9cJUoRiSPQcXmEFKR4KFKlfdr12Lv+Pp+3r9v5uFwORnTSACTO9f3tUcKfiLbOsFQsEVMxEkwtMRNROtO9b4V0yT5TAcoq5O5QBtkssSeV6LxnSUXWVEKDp5jmDocmNa3e6ZapYJE9OypCWXxXDo0BC2VJKr+qT3AdTXsialw5BHb/5ZBwbRNHWm2C51/aA3n/IwNX9+RJaveE369mcdrOmNFXScdLxKUg64GgkkqLFrz4JjJ+rj/Cx4XMxgMAAB+LCAAAAAAAAAOLrQUAZzx4YAIAAAA='''

print(gzip.decompress(base64.b64decode(base64gzip)))

^^ Python outputs this data, including the last ']}' characters.

b'{"data":[["190296311161"],\n["190296311154"],\n["190296311154"],\n["190296328299"],\n["190296328275"],\n["190296303203"],\n["190296303197"],\n["190296333002"],\n["190296303883"],\n["190296350870"],\n["190296307515"],\n["190296307164"],\n["190296309168"],\n["190296309151"],\n["190296305863"],\n["075679761255"],\n["190296303982"],\n["190296303975"],\n["190296332784"],\n["190296336621"],\n["190296336607"],\n["190296317552"],\n["190296317545"],\n["190296352591"],\n["190296352584"],\n["190296306297"],\n["190296334955"],\n["190296352263"],\n["190296352263"],\n["190296323584"],\n["190296350139"],\n["5054283041637"],\n["5054283014655"],\n["5054283014648"],\n["5054283014631"],\n["190296350146"],\n["190296306273"],\n["190296310751"],\n["190296310744"],\n["190296315992"],\n["190296315992"],\n["190296315992"],\n["190296315992"],\n["190296315992"],\n["190296315985"],\n["190296315985"],\n["190296315985"],\n["190296315985"],\n["190296315985"],\n["190296340710"],\n["5054283120622"],\n["190296305870"],\n["190296330094"]]}'

Can some one point me in the right direction why this fails in PHP and how I can still make sure I get the same content as for e.g. in Python?

gzip python php
1 Answers

Your gzip stream is valid and correct. It consists of two gzip members, each of which is a valid gzip stream on its own, the second of which contains only the the characters ]}. Python is correctly decoding both gzip members, whereas it appears that PHP is decoding only the first gzip member, ignoring and silently discarding the second.

This is a bug in PHP's gzdecode(), and appears to still be there in the latest PHP.

An alternative is to use inflate_init(), inflate_add(), and inflate_get_read_len() in a loop to read all of the members. The key is that inflate_get_read_len() is the number of bytes of the gzip stream that have been consumed so far, enabling you to find the start of the next gzip member. Here is an example I ran in a PHP playground with your data to show the idea:

$b64 = 'H4sIAAAAAAAAA7XSO04DQAwE0J5jbE3h8d9cJUoRiSPQcXmEFKR4KFKlfdr12Lv+Pp+3r9v5uFwORnTSACTO9f3tUcKfiLbOsFQsEVMxEkwtMRNROtO9b4V0yT5TAcoq5O5QBtkssSeV6LxnSUXWVEKDp5jmDocmNa3e6ZapYJE9OypCWXxXDo0BC2VJKr+qT3AdTXsialw5BHb/5ZBwbRNHWm2C51/aA3n/IwNX9+RJaveE369mcdrOmNFXScdLxKUg64GgkkqLFrz4JjJ+rj/Cx4XMxgMAAB+LCAAAAAAAAAOLrQUAZzx4YAIAAAA=';
$bin = base64_decode($b64);
$gz = inflate_init(ZLIB_ENCODING_GZIP);
$dec = inflate_add($gz, $bin);
echo inflate_get_status($gz), "\n";
echo inflate_get_read_len($gz), "\n";
echo strlen($dec), "\n";
$used = inflate_get_read_len($gz);
$gz = inflate_init(ZLIB_ENCODING_GZIP);
$dec .= inflate_add($gz, substr($bin, $used));
echo inflate_get_status($gz), "\n";
echo inflate_get_read_len($gz), "\n";
echo strlen($dec), "\n";

This outputs:

1
217
966
1
22
968

where the 1's are the expected return codes for a complete and correct gzip member (make sure to check for that), the 217 and the 22 are the lengths of the two gzip members, and the 966 and 968 are the accumulated amounts of decompressed data, with the second one showing that those errant two characters at the end were added.

You can write your own gzdecode_complete() using those functions that does not have the bug.

user_11806200
Related