How to save a very large numpy array as an image, loading as little as possible into memory

user_10127292 0
Viewed 3159

My program is often working with numpy arrays that are very large ((819200, 460800, 4), uint8). To store this in memory (as just pure zeros) I would need over 1.3TB of memory which is absurd. My goal is to be able to save these numpy arrays as an image. I also want it to be as fast as it can, but speed is less of an issue.

What I was doing at the very beginning was storing the numpy array in an HDF5 file (using H5PY), then I would do processing on that array, then use CV2 to save. Not only was it slow, CV2 seems to load the image into memory, so that idea went out of the window very quickly. Now, I have tried over 20 different methods of saving these large arrays, so to shorten the post, I'll just mention some of the latest one.

After using CV2, I found a library called 'numpngw'. Its a numpy and python based png writer. This was my code:

f = h5py.File("mytestfile.hdf5", "w")
dset = f.create_dataset("mydataset", (100000,100000,4), dtype=np.uint8, compression='gzip')

shp = dset.shape    
step = 10000


png = open("new.png", "wb")
numpngw._write_header_and_meta(png, 8, shp, color_type=6, bitdepth=8, palette=None, #i'm manually writing to the png file rather than writing all data at once, so i can append data over and over again.
                            interlace=0, text_list=None, timestamp=None, sbit=None, gamma=None, iccp=None,
                            chromaticity=None, trans=None, background=None, phys=None)


for i in range(0, shp[0]+step, step): #from step to 
    numpngw._write_data(png, dset[i:i+step, i:i+step], bitdepth=8, max_chunk_len=step, #writing the data in largest chunks I can
                    filter_type=None, interlace=0)
    png.flush()
    #gc.collect()

    numpngw._write_iend(png)

png.close()
f.close()

The idea of this is that it just writes chunks of the numpy array over and over again until the whole array is written. I never even knew if this version was working because it was so extremely slow.

I then tried the same chunk writing method with PIL. And instead of a PNG, I used TIFF since it seemed a lot faster. Unfortunately, PIL doesn't support appending to a TIFF in chunks. The "append" parameter is used for animating TIFFs, so I couldn't do it that way.

The last, and most recent library I used was tifffile. It seemed to do everything I needed. It also has a memmap implementation to make a TIFF file out of a memory mapped numpy array.

blank = numpy.zeros((256,256,3))
memmap_image = tifffile.memmap('temp.tif', shape=blank.shape, dtype='uint8')
memmap_image[:] = blank[:]
memmap_image.flush()
del memmap_image

This creates a blank TIFF file. Combining this with H5PY allowed me to save large images - or so I thought. The large TIFF files seem to be corrupt. I tried opening them in (Windows) Photos, Adobe Acrobat Reader DC and Affinity Photo. All said the file was not recognised (sometimes Affinity Photo even crashed on opening - could be a memory issue though). I don't know what would be making the image corrupt since it seems to work with smaller arrays. I came back to it the next day and started getting memory errors on this line (out of nowhere) memmap_image[:] = blank[:].

The last thing I tried was combining the chunk method with tifffile:

f = h5py.File("mytestfile.hdf5", "w")
dset = f.create_dataset("mydataset", (100000,100000,3), dtype=np.uint8)

shp = dset.shape    
step = 10000

a = tiffile.memmap('temp.tif', shape=(100000,100000,3), dtype=np.uint8)

for i in range(0, shp[0]+step, step):
    a[i:i+step,i:i+step] = dset[i:i+step,i:i+step]
    a.flush()
del a

It takes about 2 minutes (not bad!) and it created a large file (~29GB, compression will make it smaller), however, again it is corrupt and nothing can read the TIFF file.

I really don't want to give up on this project but I'm stuck on what else I can try. Could anyone recommend a TIFF/PNG library that supports appending to the image, but also doesn't want to load it into memory?

tiff memory-management python-3.x numpy arrays
3 Answers

Standard TIFF cannot be used to store a 100000x100000 RGB image unless maybe if it is extremely compressible. The size of TIFF files is limited to 4 GB due to the usage of 32-bit offsets. BigTIFF uses 64-bit offsets. To enable tifffile writing BigTIFF, use the bigtiff=True parameter with memmap or imwrite. However, not many software/libraries will be able to read the files because BigTIFF and/or large strip sizes are not supported.

Images this large are commonly stored tiled, often with compression and multiple resolutions (pyramids). Tifffile can create tiled (Big)TIFF from a memory-mapped numpy array or a generator of tiles, e.g.:

import numpy
import h5py
import tifffile

dtype = 'uint8'
shape = 100000, 100000, 3
tileshape = 1024, 1024

f = h5py.File('test.hdf5', 'w')
data = f.create_dataset('test', shape, dtype=dtype, compression='gzip')


def tile_generator(data, tileshape):
    for y in range(0, data.shape[0], tileshape[0]):
        for x in range(0, data.shape[1], tileshape[1]):
            tile = data[y: y+tileshape[0], x: x+tileshape[1], :]
            if tile.shape[:2] != tileshape:
                pad = (
                    (0, tileshape[0] - tile.shape[0]),
                    (0, tileshape[1] - tile.shape[1]),
                    (0, 0)
                )
                tile = numpy.pad(tile, pad, 'constant')
            yield tile


tifffile.imwrite(
    'temp.tif', 
    tile_generator(data, tileshape),
    dtype=dtype, 
    shape=shape, 
    tile=tileshape,
    bigtiff=True,
    # compress='jpeg'
)

tifffile supports several compression options via the imagecodecs library, e.g. DEFLATE, LZMA, ZStd, JPEG, JPEG2000, JPEGXR, WebP...

There are dedicated TIFF "sub-formats", libraries, and tools to deal with pyramidal TIFF, often depending on the application domain, e.g. libvips, OpenSlide, GDAL, or BioFormats.

user_4534630

Here's a libvips example for creating a huge TIFF file without needing loads of memory:

import pyvips

# - make a set of pyvips images from a set of pointers to memory mapped files
# - the pointer objects need to support the buffer protocol, ie. refcounts,
# and will not be copied
# - format is something like "char" or "float"
images = [pyvips.Image.new_from_memory(pointer, width, height, bands, format)
          for pointer in my_set_of_pointers]

# join into a huge image, eg. 100 tiles across
# you can set margins, alignment, spacing, background, etc.
huge = pyvips.Image.arrayjoin(images, across=100)

# write to a file ... you can set a range of options, see eg. the 
# tiffsave docs
huge.write_to-file("thing.tif", compression="jpeg", tile=True, bigtiff=True)

It can efficiently read and write TIFF pyramids, set the pyramid option. libvips 8.10 also supports bioformats pyramids.

The libvips GUI, nip2, can display images of any size, including huge bigtiffs. It might be worth trying if other viewers are struggling. I regularly work with 300,000 x 300,000 pixel images on this modest laptop.

user_8947630

Nowadays tifffile has some functions to work with very large files.

from tifffile import TiffWriter, memmap

# reading
original_image = memmap('/path/to/uncompressed/3D_image.tif')

# writing
with TiffWriter('/path/to/save/3D_image.tif', bigtiff=True) as tif:
    for i in range(rotated.shape[0]):
        tif.write(rotated[i], photometric='minisblack')
user_18114420
Related