I am trying to go through a folder full of pickle (.pkl) files and combine all the dataframes from the pickle files into one HDF5 file.
However, when I try to append dfs[] with 100 data frames, I get a "Killed: 9" error halfway through the for loop. Does anyone know a better way to go about doing this? I want to be able to have all my data in one frame, however it seems difficult since there is a lot of data.. Think 1 billion rows total.
dfs = []
count = 0
path = '/Users/srayan/Desktop/Pickle'
for filename in glob.glob(os.path.join(path, '*.pkl')):
dfs.append(pd.read_pickle(filename))
df = pd.concat(dfs)
key = 'my_key'
df.to_hdf('merged.h5', key)