How can i compress best possible 50000 bytes which are partly sorted

Viewed 162

I want compress best possible 50000 bytes, which are partly sorted.

  • That means there are 256 increasing runs of bytes like 0,0,3,4,5,6,6,9,....,250.

  • Furthermore there exists ca. 630000000 inversions. I have calculated the inversions by using bubblesort and count the decreasing pairs.

  • Each Byte is equal often present.

  • The runs.length differ in 20%

Currently I use deltacompression for the runs and huffman for the encoding and i get ca. 14000bytes out. How its possible to compress it further?

Example Link: http://pastebin.com/raw/we57yZUw File bytes comma seperated in list. No encoding.

1 Answers

Short answer, you've already come impressively close to the theoretical limit. You can't do much better unless there is some information that you have about these sequences that you haven't shared with us. And by "much better", you can't actually get to 14k, let alone 10k.

Let's start by looking at how many such sequences there could be.

We can turn your sequence into a non-decreasing sequence in the range 0-65,535 by adding 256 times the number of completed runs to each number. That is, instead of "wrapping around", you just keep climbing. This is a one-to-one correspondence, though admittedly some (actually a lot) of the increasing sequences will not correspond with sequences that you are looking to encode. Please note that fact, I will return to that later.

How many non-decreasing sequences are there of length 50,000 in the range 0-65,535 are there? Well, those sequences are in a one-to-one correspondence with sequences of 115,535 bits, 50,000 of which are zero. The correspondence is that you replace each 0 bit with a count of how many 1 bits happened before it to get numbers out of the sequence.

The number of sequences of 115,535 bits, 50,000 of which are zero, is 115,535 choose 50,000. Which is 115535! / (50000! * 65535!). This is a rather large number, but we can estimate its log using Stirling's formula that log_2(n!) = n log_2(n) - n * log_2(e) + log_2(pi * n / 2) + O(log(n)). To that end:

log_2(115535) = 16.8179704401675
log_2(65535) = 15.9999779860527
log_2(50000) = 15.6096404744368
log_2(pi * 115535 / 2) = 17.4694665696399
log_2(pi * 65535 / 2) = 16.6514741155252
log_2(pi * 50000 / 2) = 16.2611366039092
log_2(e) = 1.44269504088896

And now, please double check for calculation mistakes,

log_2(115535 choose 50000)
    = log_2(115535!) - log_2(65535!) - log_2(50000!)
    = 115535 * log_2(115535) - 115535 * log_2(e) + log_2(pi * 115535 / 2) + O(log(115535))
      - (65535 * log_2(65535) - 65535 * log_2(e) + log_2(pi * 65535 / 2) + O(log(65535)))
      - (50000 * log_2(50000) - 50000 * log_2(e) + log_2(pi * 50000 / 2) + O(log(50000)))
    = 115535 * 16.8179704401675 - 115535 * 1.44269504088896 + 17.4694665696399 + O(log(115535))
      - (65535 * 15.9999779860527 - 65535 * 1.44269504088896 + 16.6514741155252 + O(log(115535)))
      - (50000 * 15.6096404744368 - 50000 * 1.44269504088896 + 16.2611366039092 + O(log(115535)))
    = 1776399.91272222 - 954028.189285421 - 708363.532813996 + O(16.8179704401675)
    = 114008.190622803 + O(16.8179704401675)

If you wish to expand a couple of terms of the Stirling series, you'll find that 114008.190622803 is off from the true log by something less than 1.

Therefore you require somewhere around 114008 bits to specify one such sequence, and that takes 14251 bytes. Which means that if you're managing to compress it to the 14k range, you are very close to the theoretical limit. Unless you have more information about your sequences than you have provided, you can't do better.

BUT I'm cheating. You specified that the length of the runs varied by no more than about 20%. I'm including ones where the length of the runs not only varies, but even where some runs can be skipped entirely! How big a deal is that?

Well from a simulation that I just ran, a run gets within 10% of the median over 50% of the time. We have 256 runs. Their lengths are not actually independent, but long runs are correlated with short ones and vice versa, so with odds better than 1/2^256, we will randomly satisfy the length requirement. This means that the length condition saves no more than 32 bytes off of the theoretical best. Given that we're talking around 14k of data already, this isn't a significant improvement.

Related