How to Read nrows From Pandas HDF Storage?

Viewed 1346

What am I trying to do?

pd.read_csv(... nrows=###) can read the top nrows of a file. I'd like to do the same while using pd.read_hdf(...).

What is the problem?

I am confused by the documentation. start and stop look like what I need but when I try it, a ValueError is returned. The second thing I tried was using nrows=10 thinking that it might be an allowable **kwargs. When I do, no errors are thrown but also the full dataset is returned instead of just 10 rows.

Question: How does one correctly read a smaller subset of rows from an HDF file? (edit: without having to read the whole thing into memory first!)

Below is my interactive session:

>>> import pandas as pd
>>> df = pd.read_hdf('storage.h5')
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    df = pd.read_hdf('storage.h5')
  File "C:\Python35\lib\site-packages\pandas\io\pytables.py", line 367, in read_hdf
    raise ValueError('key must be provided when HDF5 file '
ValueError: key must be provided when HDF5 file contains multiple datasets.
>>> import h5py
>>> f = h5py.File('storage.h5', mode='r')
>>> list(f.keys())[0]
'table'
>>> f.close()
>>> df = pd.read_hdf('storage.h5', key='table', start=0, stop=10)
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    df = pd.read_hdf('storage.h5', key='table', start=0, stop=10)
  File "C:\Python35\lib\site-packages\pandas\io\pytables.py", line 370, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "C:\Python35\lib\site-packages\pandas\io\pytables.py", line 740, in select
    return it.get_result()
  File "C:\Python35\lib\site-packages\pandas\io\pytables.py", line 1447, in get_result
    results = self.func(self.start, self.stop, where)
  File "C:\Python35\lib\site-packages\pandas\io\pytables.py", line 733, in func
    columns=columns, **kwargs)
  File "C:\Python35\lib\site-packages\pandas\io\pytables.py", line 2890, in read
    return self.obj_type(BlockManager(blocks, axes))
  File "C:\Python35\lib\site-packages\pandas\core\internals.py", line 2795, in __init__
    self._verify_integrity()
  File "C:\Python35\lib\site-packages\pandas\core\internals.py", line 3006, in _verify_integrity
    construction_error(tot_items, block.shape[1:], self.axes)
  File "C:\Python35\lib\site-packages\pandas\core\internals.py", line 4280, in construction_error
    passed, implied))
ValueError: Shape of passed values is (614, 593430), indices imply (614, 10)
>>> df = pd.read_hdf('storage.h5', key='table', nrows=10)
>>> df.shape
(593430, 614)

Edit:

I just attempted to use where:

mylist = list(range(30))
df = pd.read_hdf('storage.h5', key='table', where='index=mylist')

Received a TypeError indicating a Fixed format store (the default format value of df.to_hdf(...)):

TypeError: cannot pass a where specification when reading from a
  Fixed format store. this store must be selected in its entirety

Does this mean I can't select a subset of rows if the format is Fixed format?

2 Answers

I ran into the same problem. I am pretty certain by now that https://github.com/pandas-dev/pandas/issues/11188 tracks this very problem. It is a ticket from 2015 and it contains a repro. Jeff Reback suggested that this is actually a bug, and he even pointed us towards a solution back in 2015. It's just that nobody built that solution yet. I might have a try.

Seems like this now works, at least with pandas 1.0.1. Just provide start and stop arguments:

df = pd.read_hdf('test.h5', '/floats/trajectories', start=0, stop=5)
Related