How to generate a sample of x and y with conditional probability with NumPy

user_11079284 0
Viewed 865

I'm trying to generate a sample of x's and their labels - y's, for binary classifier.

I know that my x's are uniformly distributed in [0,1]. But my y's distribution derived by my x's: if x in [0.2, 0.4] or in [0.6, 0.8] - P[Y=1] = 0.1. If x is outside of these bounds then P[Y=1] = 0.8 .

I think that the best way to do that is by using NumPy (and not using for-loops and if-condition) but until now I didn't succeed.

This is my attempt:

s = np.random.uniform(0,1,100) # 100 x samples in [0,1] uniformly distributed
condition  = (np.logical_or((s>0.2)&(s < 0.4), (s>0.6)&(s < 0.8))) # attempt to mark with True the places of x in bounds.
x_in_bounds = np.select(condlist, s) # this line doesn't work
... # how to generate the y values?

I am trying unsuccessfully to find a way to randomly generate the y values according to the conditions on the sample of the x values. I'd love to understand what I'm missing.

numpy-ndarray numpy python
3 Answers

For a solution using your approach see @adnanmuttaleb 's answer.

My approach for this would be using numpy's advanced indexing:

x = np.random.uniform(0, 1, 100)

cond = ((x > 0.2) & (x < 0.4)) | ((x > 0.6) & (x < 0.8))
not_cond = np.logical_not(cond)

y = np.random.rand(*x.shape)
y[cond] = y[cond] < 0.1
y[not_cond] = y[not_cond] < 0.8
y = y.astype(int)
user_34562620

One way could be to generate two random sequences, filled with either 1 or 0 according to both mentioned cases. Then use np.where to select from one or the other depending on condition:

s = np.random.uniform(0,1,100)
condition  = np.logical_or((s>0.2)&(s < 0.4), (s>0.6)&(s < 0.8))

repl_a = (np.random.random(len(s))>0.9).view('i1')
repl_b = (np.random.random(len(s))>0.2).view('i1')

np.where(condition, repl_a, repl_b)

array([1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1,
       0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0,
       1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1,
       0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0], dtype=int8)
user_96986840

A solution using the same approach you are using, would be: 

generate = lambda prob: 1 if np.random.rand() < prob else 0

s = np.random.uniform(0, 1, 100)
low_prob_condition = ((s > 0.2) & (s < 0.4)) | ((s > 0.6) & (s < 0.8))
condlist = [low_prob_condition, np.logical_not(low_prob_condition)] 
labels = np.select(condlist, [[generate(0.1) for _ in range(s.size)], [generate(0.8) for _ in range(s.size)]])

print(labels)

Output:

[1 1 0 1 0 1 0 1 0 1 0 1 0 1 1 0 0 1 1 1 1 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 0
 1 1 0 0 1 0 1 1 1 0 0 1 1 0 0 1 0 1 0 0 0 0 1 0 0 1 0 1 1 0 1 0 0 0 1 0 0
 0 0 1 0 1 0 0 0 0 0 1 0 1 0 0 0 1 1 1 1 0 1 0 0 1 1]

But more time and space efficent solution whould be:

s = np.random.uniform(0, 1, 100)
low_prob_cond = lambda x: ((x > 0.2) and (x < 0.4)) or ((x > 0.6) and (x < 0.8))
gen = lambda prob: 1 if np.random.rand() < prob else 0
labels = (gen(0.1) if low_prob_cond(x) else gen(0.8) for x in s)

print(list(labels))

Output: 

[0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1]

np.select requries a list of same size with s as choice list for each condition (in your case two), this is obviously can be avoided in your problem.

user_35792120
Related