Python-Generating numbers according to a corellation matrix

user_8041133 0
Viewed 1755

enter image description hereenter image description here

enter image description here

Hi, I am trying to generate correlated data as close to the first table as possible (first three rows shown out of a total of 13). The correlation matrix for the relevant columns is also shown (corr_total).

I am trying the following code, which shows the error: "LinAlgError: 4-th leading minor not positive definite" 

from scipy.linalg import cholesky

# Correlation matrix

# Compute the (upper) Cholesky decomposition matrix

upper_chol = cholesky(corr_total)

# What should be here? The mu and sigma of one row of a table?
rnd = np.random.normal(2.57, 0.78, size=(10,7))


# Finally, compute the inner product of upper_chol and rnd
ans = rnd @ upper_chol

My question is what goes into the values of The mu and sigma, and how to resolve the error shown above. Thanks! P.S I have edited the question to show the original table. It shows data for four patients. I basically want to make synthetic data for more cases, that replicates the patterns found in these patients

correlation python
1 Answers

Thank you for answering my question about when data you have access to. The error that you received was generated when you called cholesky. cholesky requires that your matrix be positive semidefinite. One way to check if a matrix is semi-positive definite is to see if all of its eigenvalues are greater than zero. One of the eigenvalues of your correlation/covarance matrix is nearly zero. I think that cholesky is just being fussy. Use can use scipy.linalg.sqrtm as an alternate decomposition.

For your question on the generation of multivariate normals, the random normal that you generate should be a standard random normal, i.e. a mean of 0 and a width of 1. Numpy provides a standard random normal generator with np.random.randn. To generate a multivariate normal, you should also take the decomposition of the covariance, not the correlation matrix. The following will generate a multivariate normal using an affine transformation, as in your question.

from scipy.linalg import cholesky, sqrtm
relavant_columns = ['Affecting homelife',
           'Affecting mobility',
           'Affecting social life/hobbies',
           'Affecting work',
           'Mood',
           'Pain Score',
           'Range of motion in Doc']

# df is a pandas dataframe containing the data frame from figure 1
mu = df[relavant_columns].mean().values
cov = df[relavant_columns].cov().values
number_of_sample = 10


# generate using affine transformation
#c2 = cholesky(cov).T
c2 = sqrtm(cov).T
s = np.matmul(c2, np.random.randn(c2.shape[0], number_of_sample)) + mu.reshape(-1, 1)

# transpose so each row is a sample
s = s.T

Numpy also has a built-in function which can generate multivariate normals directly

s = np.random.multivariate_normal(mu, cov, size=number_of_sample)
user_50786980
Related