How can I get the "shape" of some data so I can generate similar random numbers in numpy/scipy

user_1171112 0
Viewed 316

Apologies. I know what I want to do, but am not sure what it is called and so haven't been able to search for it.

I am chasing down some anomalies in data (two reports which should add to the same total based on about 50K readings differ slightly). I therefore want to generate some random data which is the same "shape" as the data in question in order to determine whether this might be down to rounding error.

Is there a way of analysing the existing 50K or so numbers and then generating random numbers which would look pretty much the same shape on a histogram? My presumption is that numpy is probably the best tool for this, but I am open to advice.

random numpy python
2 Answers

You can use scipy's stats package to do this, if I'm interpreting your question correctly:

First, we generate a histogram, and measure its histogram distribution using the scipy.stats.rv_histogram() method

import scipy.stats
import numpy as np
import matplotlib.pyplot as plt

data = scipy.stats.norm.rvs(size=50000, loc=0)
hist = np.histogram(data, bins=100)
dist = scipy.stats.rv_histogram(hist)

To generate new data from this histogram, we simply call the rvs() method on the dist variable:

fake_data = dist.rvs(size=50000)

Then, we show the two distributions to prove we are getting what we expect:

plt.figure()
plt.hist(data,bins=100, alpha=0.5, label='real data')
plt.hist(fake_data,bins=100, alpha=0.5, label='fake data')
plt.legend(loc='upper right')
plt.show()

enter image description here

Hopefully this is what you're looking to do.

user_137409910
user_29772560
Related