How much space needed for a R matrix cell with 0 value? And how to deal with big matrix calculation

user_3179698 0
Viewed 46

I want to calculate the correlation for a matrix, but stopped by program since the memory is not able to fit.

M = matrix(0, length(user), length(mat))

when I run this line,I got a message

Error: cannot allocate vector of size 146487.5 Gb

FYI

length(user) = 6728367
length(mat) = 2922138

so how much space a 0 value matrix cell need to occupy?

And further more, how to deal with the matrix calculation based on the sparse matrix? Is there a package? or I can do a multiple core calculation?

sparse-matrix r memory-management matrix multithreading
1 Answers

How much space is needed is mentioned in your error message: 146,487.5 GBs.

If you want such a big matrix, just use a sparse matrix:

library(Matrix)
user = 6728367
mat = 2922138
M = Matrix(0, user, mat)

This will work:

> str(M)
Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
  ..@ i       : int(0) 
  ..@ p       : int [1:2922139] 0 0 0 0 0 0 0 0 0 0 ...
  ..@ Dim     : int [1:2] 6728367 2922138
  ..@ Dimnames:List of 2
  .. ..$ : NULL
  .. ..$ : NULL
  ..@ x       : num(0) 
  ..@ factors : list()

There is a tutorial here and you can find a lot more online. Mathematical calculations are optimised with the package, so something like M * M will work fast as well. Also, you can check here

user_41300440
Related