Is there a way to create correlated variables in Python setting the standard deviation of these variables?

user_16175033 0
Viewed 357

I want to create fake data for four variables: Height, Weight, Age and Income.

I have use this script:

cov_matrix = [[1, 0.7, 0, 0],
              [0.7, 1, 0, 0],
              [0, 0, 1, 0.4],
              [0, 0, 0.4, 1]]
correlated = np.random.multivariate_normal([165, 65, 30, 15000], cov_matrix, size=250)
data = pd.DataFrame({
 "Height": correlated[:, 0],
 "Weight": correlated[:, 1],
 "Age": correlated[:, 2],
 "Income": correlated[:, 3]
})

But the results are not good enough, the standard deviation (sd) of the four variables is approximately 1, and I want my data to have more dispersion. For example, being 30 the sd for the variable "Height".

Is there any possibility to achieve this in Python?

correlation python
1 Answers

To get variances of each features, simply put those values in the diagonal of the covariance matrix. The off-diagonal elements need to be scaled to account for the feature variances however.

a1 = 0.7*np.sqrt(30*12)
a2 = 0.4*np.sqrt(19*50)
cov_matrix = np.array([[30.0,   a1,  0.0, 0.0],
                       [  a1, 12.0,  0.0, 0.0],
                       [ 0.0,  0.0, 19.0, a2],
                       [ 0.0,  0.0,   a2, 50.0]])

correlated = np.random.multivariate_normal([165, 65, 30, 15000], cov_matrix, size=1000)
print(correlated.var(axis=0))
print(np.corrcoef(correlated.T))

Variances:

[28.02834149 11.14644597 18.68960579 49.46234297]

Cross-correlation coefficient matrix between features:

[[ 1.          0.67359842 -0.02016119 -0.02607946]
 [ 0.67359842  1.         -0.00338224 -0.01021924]
 [-0.02016119 -0.00338224  1.          0.37187791]
 [-0.02607946 -0.01021924  0.37187791  1.        ]]

Alternatively, generate the data according to the original covariance matrix and then scale and shift each of the features to have the desired mean and standard deviation. This will preserve the correlation coefficient as originally intended. Note that the mean is added after scaling because otherwise the scaling will change the mean.

# generate correlated features with zero-mean and unit variance
correlated = np.random.multivariate_normal(np.zeros(4), cov_matrix, size=1000)

# multiply by the desired standard deviation to scale the data and add the mean
correlated = correlated.dot(np.diag(np.sqrt([30.0, 12.0, 19.0, 50.]))) + np.array([165, 65, 30, 15000])
user_168797940
Related