react-virtulized list with ArrowKeyStepper not working

I am trying to get my react-virtualized List component to work together with ArrowKeyStepper without success. I must be missing something but could not find any examples of this implementation. There are no errors it just does not function as expected.

So far my code looks like this:

import { List as VirtualizedList, AutoSizer, ArrowKeyStepper } from "react-virtualized";
...skip a lot of other stuff
<div data-testid="popover-items">
          <ArrowKeyStepper columnCount={1} rowCount={numFilteredOptions} mode="cells">
            {({ onSectionRendered, scrollToRow, scrollToColumn }) => (
              <AutoSizer disableHeight={!fullScreen}>
                {({ width, height }) => (
                  <VirtualizedList
                    rowHeight={rowHeight}
                    height={fullScreen ? height - 175 : VIRTUAL_LIST_HEIGHT}
                    onRowsRendered={onSectionRendered}
                    columnCount={1}
                    rowCount={numFilteredOptions}
                    rowRenderer={renderRow}
                    noRowsRenderer={() => null}
                    scrollToRow={scrollToRow}
                    scrollToColumn={scrollToColumn}
                    width={width}
                  />
                )}
              </AutoSizer>
            )}
          </ArrowKeyStepper>
        </div>

Expected Behavior: arrow up/down would move the selection of an item in the list.

Actual Behavior: nothing happens.