Need an algorithm to "evenly" iterate over all possible combinations of a set of values

Viewed 374

sorry for the horrible title, I am really struggling to find the right words for what I am looking for. I think what I want to do is actually quite simple, but I still can't really wrap my head around creating algorithms. I bet I could have easily found a solution on the web if I wasn't lacking basic knowledge of algorithm terminology.

Let's assume I want to iterate over all combinations of an array of five integers, where each integer is a number between zero and nine. Naturally, I could just increment from 0 to 99999. [0, 0, 0, 0, 1], [0, 0, 0, 0, 2], ... [9, 9, 9, 9, 9].

However, I need to "evenly" (don't really know how to call it) increment the individual elements. Ideally, the sequence of arrays that is produced by the algorithm should look something like this:

[0,0,0,0,0] [1,0,0,0,0] [0,1,0,0,0] [0,0,1,0,0] 
[0,0,0,1,0] [0,0,0,0,1] [1,1,0,0,0] [1,0,1,0,0] 
[1,0,0,1,0] [1,0,0,0,1] [1,1,0,1,0] [1,1,0,0,1]
[1,1,1,0,0] [1,1,1,1,0] [1,1,1,0,1] [1,1,1,1,1] 
[2,0,0,0,0] [2,1,0,0,0] [2,0,1,0,0] [2,0,0,1,0] 
[2,0,0,0,1] [2,1,1,0,0] [2,1,0,1,0] ..... 

I probably made a few mistake in the sequence above, but maybe you can guess what I am trying to approach. Don't introduce a number higher than 1 unless every possible combination of 0s and 1s has been determined, don't introduce a number higher than 2 unless every possible combination of 0s, 1s and 2s has been determined, and so on..

I would really appreciate someone pointing me in the right direction! Thanks a lot

2 Answers

You could break this down into two subproblems:

  • get all combinations with replacement of 0, 1, 2, ... for the given number of digits
  • get all (unique) permutations of those combinations

Your desired ordering is still different than the order those are typically generated in (e.g. (0,1,1) before (0,0,2), and (0,0,1) before (1,0,0)), but you can just collect all the combinations and all the permutations individually and sort them, at least requiring much less memory than for generating, collecting and sorting all those combinations.

Example in Python, using implementations of those functions from the itertools library; key=lambda c: c[::-1] sorts the lists in-order, but reversing the order of the individual elements to get your desired order:

from itertools import combinations_with_replacement, permutations

places = 3
max_digit = 3

all_combs = list(combinations_with_replacement(range(0, max_digit+1), r=places))
for comb in sorted(all_combs, key=lambda c: c[::-1]):
    all_perms = set(permutations(comb))
    for perm in sorted(all_perms, key=lambda c: c[::-1]):
        print(perm)

And some selected output (64 elements in total)

(0, 0, 0)
(1, 0, 0)
(0, 1, 0)
...
(0, 1, 1)
(1, 1, 1)
(2, 0, 0)
(0, 2, 0)
...
(0, 1, 2)
(2, 1, 1)
...
(2, 2, 2)
(3, 0, 0)
(0, 3, 0)
...
(2, 3, 3)
(3, 3, 3)

For 27 places with values up to 27 that would still be too many combinations-with-replacement to generate and sort, so this part should be replaced with a custom algorithm.

  • keep track of how often each digit appears; start with all zeros
  • find the smallest digit that has a non-zero count, increment the count of the digit after that, and redistribute the remaining smaller counts back to the smallest digit (i.e. zero)

In Python:

def generate_combinations(places, max_digit):
    # initially [places, 0, 0, ..., 0]
    counts = [places] + [0] * max_digit
    yield [i for i, c in enumerate(counts) for _ in range(c)]
    while True:
        # find lowest digit with a smaller digit with non-zero count
        k = next(i for i, c in enumerate(counts) if c > 0) + 1
        if k == max_digit + 1:
            break
        # add one more to that digit, and reset all below to start
        counts[k] += 1
        counts[0] = places - sum(counts[k:])
        for i in range(1, k):
            counts[i] = 0
        yield [i for i, c in enumerate(counts) for _ in range(c)]

For the second part, we can still use a standard permutations generator, although for 27! that would be too many to collect in a set, but if you expect the result in the first few hundred combinations, you might just keep track of already seen permutations and skip those, and hope that you find the result before that set grows too large...

from itertools import permutations

for comb in generate_combinations(places=3, max_digit=3):
    for p in set(permutations(comb)):
        print(p)
    print()

You've already said that you can get the combinations you are looking for by enumerating all nk possible sequences, except that you don't get them in the desired order.

You could generate the sequences in the right order if you used an odometer-style enumerator. At first, all digits must be 0 or 1. When the odometer would wrap (after 1111...), you increment the set of the digits to [0, 1, 2]. Reset the sequence to 2000... and keep iterating, but only emit sequences that have at least one 2 in them, because you've already generated all sequences of 0's and 1's. Repeat until after wrapping you go beyond the maximum threshold.

Filtering out the duplicates that don't have the current top digit in them can be done by keeping track of the count of top numbers.

Here's an implementation in C with hard-enumed limits:

enum {
    SIZE = 3,
    TOP = 4
};

typedef struct Generator Generator;

struct Generator {
    unsigned top;           // current threshold
    unsigned val[SIZE];     // sequence array
    unsigned tops;          // count of "top" values
};



/*
 *      "raw" generator backend which produces all sequences
 *      and keeps track of how many top numbers there are
 */
int gen_next_raw(Generator *gen)
{
    int i = 0;
    
    do {
        if (gen->val[i] == gen->top) gen->tops--;
        gen->val[i]++;
        if (gen->val[i] == gen->top) gen->tops++;
        
        if (gen->val[i] <= gen->top) return 1;

        gen->val[i++] = 0;
    } while (i < SIZE);
   
    return 0;
}

/*
 *      actual generator, which filters out duplicates
 *      and increases the threshold if needed
 */
int gen_next(Generator *gen)
{
    while (gen_next_raw(gen)) {
        if (gen->tops) return 1;
    }
        
    gen->top++;
    
    if (gen->top > TOP) return 0;
    
    memset(gen->val, 0, sizeof(gen->val));
    gen->val[0] = gen->top;
    gen->tops = 1;    
    
    return 1;
}

The gen_next_raw function is the base implementation of the odometer with the addition of keeping a count of current top digits. The gen_next function uses it as backend. It filters out the duplicates and increases the threshold as needed. (All that can probably be done more efficiently.)

Generate the sequence with:

Generator gen = {0};

while (gen_next(&gen)) {
    if (is_good(gen.val)) {
        puts("Bingo!");
        break;
    }        
}
Related