I'm using jupyter notebook on a server (the folders are not on my computer). I have folder with 30 dataframes pickled that have exactly the same columns. They are all saved in the next path:
Reut/folder_no_one/here_the_files_located
I want to open them all and to concat them. I know I could do something like this:
df1=pd.read_pickle('table1')
df2=pd.read_pickle('table2')
df3=pd.read_pickle('table3')
...
#and then concat
but i'm sure there is better and smarters way to do so. I have tried to open all the files and save them seperatly as following:
num=list(range(1, 33)) #number of tables I have in the folder
path_to_files=r'Reut/here_the_files_located'
Path=r'Reut/folder_no_one/here_the_files_located'
{f"df{num}" : pd.read_pickle(file) for num, file in enumerate(Path(path_to_files).glob('*.pickle'))}
but I get error with this:
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) in ----> 1 {f"df{num}" : pd.read_pickle(file) for num, file in enumerate(Path(path_to_files).glob('*.pickle'))}
TypeError: 'str' object is not callable
I have tried to play and o put different version of the path, also not to put path (because my notebook is where those files are) but I keep getting the same error.
*it's important to mention that when I can open those files without specifiying the path when the notebook is inside that folder as well.
My end goal is to open and concat all those tables as one big table automatically.
edit: I have also tried this:
path = r'file_name/file_location_with_all_pickles'
all_files = glob.glob(path + "/*.pkl")
li = []
for filename in all_files:
df = pd.read_pickle(filename)
li.append(df)
frame = pd.concat(li, axis=0, ignore_index=True)
and also
path_to_files = r'file_name/file_location_with_all_pickles'
tables = []
for table in pathlib.Path(path_to_files).glob("*.pkl"):
print(table)
tables.append(pd.read_pickle(table))
but both cases I get error
ValueError: No objects to concatenate when I try to concat. also when I tell it to print the filename/table it does nothing. also if inside the loop I try to print just ordinary string (like print('hello'), nothing happens. it seems like there is problem with the path but when I open one specific pickle like this:
pd.read_pickle(r'file_name/file_location_with_all_pickles/specific_table.pkl')
it opens.
'update:
this worked for me inthe end:
import pandas as pd
import glob
path = r'folder' # use your path
all_files = glob.glob(path + "/*.pkl")
li = []
for filename in all_files:
df = pd.read_pickle(filename)
li.append(df)
frame = pd.concat(li, axis=0, ignore_index=True)
from here (Open multiple pickle files from Jupyter notebook folder doesn't work)