The following code ( specifically as.matrix ) fails only when opening very large libsvm files. It works fine on smaller files
rawmforCluster=read.matrix.csr(filePath)
sparseforCluster=rawmforCluster$x
str(sparseforCluster)
sparseMatrixforCluster=as.matrix(sparseforCluster)
The structure of sparseforCluster is
Formal class 'matrix.csr' [package "SparseM"] with 4 slots
..@ ra : num [1:4860285] 1 1 2 1 1 1 1 1 1 1 ...
..@ ja : int [1:4860285] 77 668 716 1086 1202 1306 1527 2184 2545 2729 ...
..@ ia : int [1:659095] 1 18 25 26 31 36 52 59 67 72 ...
..@ dimension: int [1:2] 659094 3778
The error I get is
Error in double(nrow * ncol) : vector size cannot be NA In addition: Warning message: In nrow * ncol : NAs produced by integer overflow
Question How do I coerce the data into a matrix or (second best) a data.table? (or should I be seeking another solution?)
Update I have found that the standard solution is to reduce the size of the matrix by removing sparse (low frequency) terms. This is not an option in my case as some low frequency terms may be highly relevant to some subsets.
I have also read about the bigmemory package. However, this does not seem to work with matrix.csr