Pythonic way of binning data without pandas/numpy

Viewed 903

I'm looking for a way to bin a dataset of several hundred entries into 20 bins. But without the use of big modules like pandas (cut) and numpy (digitize). Can anyone think of a better solution than 18 elifs?

1 Answers

All you need to do is figure out which bin each element is in. That is fairly trivial given the size of bins, if they're uniform. From your array, you can find the minval and maxval. Then, binwidth = (maxval - minval) / nbins. For an element of your array elem, and a known minimum value minval and bin width binwidth, the element will fall in bin number int((elem - minval) / binwidth). This leaves the edge case where elem == maxval. In this case, the bin number is equal to nbins (the nbins + 1th bin, because python is zero-based), so we have to decrement the bin number for just that one case.

So we can write a function that does this:

import random

def splitIntoBins(arr, nbins, minval=None, maxval=None):
    minval = min(arr) if minval is None else minval # Select minval if specified, otherwise min of data
    maxval = max(arr) if maxval is None else maxval # Same for maxval
    
    binwidth = (maxval - minval) / nbins # Bin width
    allbins = [[] for _ in range(nbins)] # Pre-make a list-of-lists to hold values

    for elem in arr:
        binnum = int((elem - minval) // binwidth) # Find which bin this element belongs in
        binindex = min(nbins-1, binnum) # To handle the case of elem == maxval
        allbins[binindex].append(elem) # Add this element to the bin
    return allbins

# Make 1000 random numbers between 0 and 1
x = [random.random() for _ in range(1000)]

# split into 10 bins from 0 to 1, i.e. a bin every 0.1
b = splitIntoBins(x, 10, 0, 1)

# Get min, max, count for each bin
counts = [(min(v), max(v), len(v)) for v in b]
print(counts)

This gives

[(0.00017731201786974626, 0.09983758434153, 101),
 (0.10111204267013452, 0.19959594179848794, 97),
 (0.20089309189822557, 0.2990120768922335, 100),
 (0.3013915797055913, 0.39922131591077614, 90),
 (0.4009006835799309, 0.49969892298935836, 83),
 (0.501675740585966, 0.5999729295882031, 119),
 (0.6010149249108184, 0.7000366124696699, 120),
 (0.7008002068562794, 0.7970568220766774, 91),
 (0.8018697850229161, 0.8990963218226316, 99),
 (0.9000732426223624, 0.9967964437788829, 100)]

Which looks like what we expect.

For non-uniform bins, it is no longer an arithmetic calculation. In this case, the element elem is in the bin that has a lower bound smaller than elem, and the an upper bound larger than elem.

def splitIntoBins2(arr, bins):
    binends = bins[1:]
    binstarts = bins[:-1]
    allbins = [[] for _ in binends] # Pre-make a list-of-lists to hold values

    for elem in arr:
        for i, (lower_bound, upper_bound) in enumerate(zip(binstarts, binends)):
            if upper_bound >= elem and lower_bound <= elem:
                allbins[i].append(elem) # Add this element to the bin
                break
    return allbins
Related