r: very large matrix.csr to matrix: Integer Overflow

Viewed 274

The following code ( specifically as.matrix ) fails only when opening very large libsvm files. It works fine on smaller files

rawmforCluster=read.matrix.csr(filePath)
sparseforCluster=rawmforCluster$x
str(sparseforCluster)
sparseMatrixforCluster=as.matrix(sparseforCluster)

The structure of sparseforCluster is

Formal class 'matrix.csr' [package "SparseM"] with 4 slots
  ..@ ra       : num [1:4860285] 1 1 2 1 1 1 1 1 1 1 ...
  ..@ ja       : int [1:4860285] 77 668 716 1086 1202 1306 1527 2184 2545 2729 ...
  ..@ ia       : int [1:659095] 1 18 25 26 31 36 52 59 67 72 ...
  ..@ dimension: int [1:2] 659094 3778

The error I get is

Error in double(nrow * ncol) : vector size cannot be NA In addition: Warning message: In nrow * ncol : NAs produced by integer overflow

Question How do I coerce the data into a matrix or (second best) a data.table? (or should I be seeking another solution?)

Update I have found that the standard solution is to reduce the size of the matrix by removing sparse (low frequency) terms. This is not an option in my case as some low frequency terms may be highly relevant to some subsets.

I have also read about the bigmemory package. However, this does not seem to work with matrix.csr

0 Answers
Related