I want to chunk over a geopandas dataframe in order to limit peak memory. Unfortunately geopandas.read_file doesn't have a chunksize option like pandas.read_csv does.
Therefore I've been using the rows=slice(start, end) option of read_file in a while loop incrementing start and end until I run out of rows. The rub is that it looks like it re-reads the whole file with each call:
In [1]: import geopandas as gpd
In [2]: fulldata = gpd.read_file('data.gpkg')
In [3]: len(fulldata)
Out[3]: 127433
In [4]: %timeit gpd.read_file('data.gpkg', rows=slice(0, 100))
52 ms ± 1.22 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [5]: %timeit gpd.read_file('data.gpkg', rows=slice(127333, 127433))
1.14 s ± 17.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Reading 100 rows at the end of the file takes 21+ times longer than the beginning. As I chunk through the file I am spending most of my time re-reading parts of the file over and over. In pandas.read_csv the chunksize option works by keeping the file pointer open and stepping through the file in order.
Is there a way to chunk through a gpkg file in geopandas that efficient?