I have a large, sparse adjacency matrix (64948 x 64948) that is symmetrical across the diagonal. What I need to do is randomize the locations of the nonzero elements in the upper or lower triangle of the matrix (which will then get transposed.) I have code that does this below. It works with 10x10 matrices, but not with 64948x64948 (I run into memory errors on a cluster). I realize my method might be flawed, and I would appreciate if anyone has any insight about how I could do this in a more efficient way!
First, I create "mask," which is essentially an array of every location in the lower triangle.
mask_mtx = np.ones([10,10]) #all ones
mask_mtx = np.tril(mask_mtx,-1) #lower triangle ones
mask_mtx = sparse.csr_matrix(mask_mtx)
mask = sparse.find(mask_mtx) #indices of ones
np.save('struc_conn_mat_mask.npy',mask) #cluster fails here when n=64948. I'm trying to save it out from a cluster so I can use the mask on my local machine with code below
len_mask = len(mask[0]) #how many indices there are
I create mtx as an array for the purposes of this example, but usually I will be reading in a 65k x 65k csr_matrix. I then find the number of nonzero elements in the mtx lower triangle and randomly pick that many locations from the mask. I then put 1s at these locations in an empty tmp_mtx. Finally, I transpose the lower triangle to upper triangle.
mtx = sparse.random(10,10,format='csr',density=0.1) #for the purposes of this example, create random matrix
lmtx = sparse.tril(mtx,-1,format='csr') #lower triangle
tmp_mtx = np.zeros((10,10)) #empty lower triangle to set
lvals = sparse.csr_matrix.count_nonzero(lmtx) #how many 1s in lmtx?
coordinate_indices = random.sample(range(len_mask),lvals) #choose n=lvals random indices to fill with ones
for idx in coordinate_indices:
tmp_mtx[mask[0][idx]][mask[1][idx]] = 1 #at randomly chosen index from mask, put a 1
tmp_mtx = sparse.csr_matrix(tmp_mtx)
mtx = tmp_mtx + tmp_mtx.T #transpose to upper triangle
Again, this works fine with 10x10 matrices, but fails at several places with larger matrices. Ultimately, what I want to do is a seemingly simple operation--shuffle the triangle--but I can't think of how to do it in a more efficient way. Perhaps there is some way of shuffling the columns and rows (but just for one of the triangles?)
Any help would be so, so appreciated! Thank you.