How to limit RAM usage of python GDAL VRT reading

user_14180970 0
Viewed 663

When reading in areas of a VRT with the python gdal library, RAM usage keeps increasing up to about 50% of available memory. This is fine on a normal computer but becomes a problem for when running on a computing cluster with huge amounts of RAM available.

Is there a way to limit how much RAM gdal uses?

Edit:

I am reading in blocks of 256x256pixels at a time with vrt.ReadAsArray(...) which are immediately used and not needed afterwards anymore. However, judging by the memory consumption, gdal is keeping read tiles in memory in case they are needed again until the available memory is about 50% filled. Only then does it start deleting unused tiles from RAM. No matter what hardware I run the program on, memory consumption will keep increasing over time until it reaches the 50% mark.

I would like to limit this to something like 32Gb RAM.

I have found a CHACHE_MAX config option of gdal. However, upon checking the amount of used cache with gdal.GetCacheUsed() it is apparently always 0. So while the option sounded promising, this does not seem to provide a solution.

gdal memory python
1 Answers

I finally did some tests and found a solution in case anyone else comes across this problem.

Although gdal.GetCacheUsed() always returned 0, changing the CACHE_MAX config option solved the problem for me. This can be set in python like this:

from osgeo import gdal
gdal.SetCacheMax(134217728) # 134Mb

While I couldn't figure out how exactly this limit applies, the cache size appears to be per band, per raster in VRT, per VRT, per process. That is, the memory usage will be higher for VRTs with many rasters and bands etc.

user_141809700
Related