Reading large .mat file in R

user_13300637 0
Viewed 181

I am trying to read a large matrix into R. The matrix has dimensionality: 3'987'288 x 93 and is about 3GB large. (Class = double) It is saved as a .mat file (it is not a v7.3 .mat file)

I tried to read the matrix with the R.matlab package:

require(R.matlab) 
A <- readMat('dat_2imp.mat')

However, it results into this error:

Error in seq.int(from = rawBufferOffset + 1L, to = rawBufferOffset + nbrOfBytes,  :
  wrong sign in 'by' argument

I did not find valuable information on this error message. However, I assume it might be due to the large size of the matrix. Do somebody know how to tackle this problem? Sadly I am not allowed to share the original data and make it fully reproducible.

The error should be raised from the function in line 88 in the code of readMat:

# Access source code:
View(R.matlab:::readMat.default)

# In line 88 defined function: 
    readRawBuffer <- function(nbrOfBytes) {
        nTotal <- length(rawBuffer)
        if (nTotal == 0L) {
            return(raw(0L))
        }
        idxs <- seq.int(from = rawBufferOffset + 1L, to = rawBufferOffset + 
            nbrOfBytes, by = 1L)
        rawBuffer[idxs]
    }

For me the defined "by" is legit and therefore the error does not make sense to me?

r matlab
1 Answers

Apparently V7.3 is not the only incompatible mat file version. As stated in the RDocuments,

readMat: Reads a MAT file structure from a connection or a file Description

Reads a MAT file structure from a connection or a file. Both the MAT version 4 and MAT version 5 file formats are supported.

And long story short, version 4 and version 5 cannot save such large dataset in one file. I think at least 2 solutions are straight forward:

  1. Exchange data in a different file format, e.g. HDF5 or SQlite. Such files are well supported both in R and matlab and do not have the compatibility issue.

  2. Save mat file in matlab in version 4 with the '-v4' switch, but there is an upper size limit in version 4, so you'll likely need to split you data across multiple files.

user_13212470
Related