I have a processing script that is designed to pull in binary data files of type "uint16" and do various processing in chunks of 6400 at a time. The code was originally written in Matlab, but because the analysis codes are written in Python we wanted to streamline the process by having everything done in Python. The problem is i've noticed that my Python code is sufficiently slower than Matlab's fread function.
Simply put the Matlab code is thus:
fid = fopen(filename);
frame = reshape(fread(fid,80*80,'uint16'),80,80);
While my Python code is simply:
with open(filename, 'rb') as f:
frame = np.array(unpack("H"*6400, f.read(12800))).reshape(80, 80).astype('float64')
The file size varies heavily from 500 MB -> 400 GB so i believe finding a faster way of parsing the data in Python could pay dividends on the larger files. A 500 MB typically has ~50000 chunks and this number increases linearly with file size. The speed difference i am seeing is roughly:
Python = 4 x 10^-4 seconds / chunk
Matlab = 6.5 x 10^-5 seconds / chunk
The processing shows over time Matlab is ~5x faster than Python's method i've implemented. I have explored methods such as numpy.fromfile and numpy.memmap, but because these methods require opening the entire file into memory at some point, it limits the use case as my binary files are quite large. Is there some pythonic method for doing this that i am missing? I would have thought Python would be exceptionally fast at opening + reading binary files. Any advice is greatly appreciated.