open, apply function and merge different files from a folder?

Viewed 22

so here's my problem:

a have the following architecture:

Repository: 
      -Sample A folder:
                       -file_A_quant.csv
                       -file_A_isdb.tsv
                       -WORKSPACE_A folder:
                                    -file_A_sir.tsv
      -Sample B folder:
                       -file_B_quant.csv
                       -file_B_isdb.tsv
                       -WORKSPACE_B folder:
                                    -file_B_sir.tsv
       -Sample N folder:
                       -file_N_quant.csv
                       -file_N_isdb.tsv
                       -WORKSPACE_N:
                                    -file_N_sir.tsv

I am able to apply a particular function to certain of the files in each folder, they're recognized by a particular suffix. i do this using this kind of function:

def ind_quant_table(repository_path, quant_suffix):

for r, d, f in os.walk(repository_path):
    for file in (f for f in f if f.endswith(quant_suffix)):
            
            complete_file_path =r+'/'+file 
            df = pd.read_csv(complete_file_path)

            df.rename(columns = lambda x: x.replace(' Peak area', ''),inplace=True)
            df.rename(columns = lambda x: x.replace(file_extention, ''),inplace=True)
            df.drop(list(df.filter(regex = 'Unnamed:')), axis = 1, inplace = True)
            df.sort_index(axis=1, inplace=True)

            prefix = 'treated_'
            df.to_csv(r+'/'+prefix+file, sep =',')

basically, with the function, I process the data in the way I want and then save it as a new file.

Now, in the same way i treat the other files, and i get individual files.

HOW can i, for each sample folder, treat the individual files and merge them and then save one unique file?. The files shared a common column that can be used to merge the information.

so far i tried something like this, but it does not work:

def ind_quant_table_f(repository_path, quant_suffix, isdb_suffix, sir_suffix):

for r, d, f in os.walk(repository_path):
    for file in (f for f in f if f.endswith(quant_table_suffix)):
            
            complete_file_path =r+'/'+file 
            df = pd.read_csv(complete_file_path)
            'do something to the df'
            df 

 for r, d, f in os.walk(repository_path):
      for file in (f for f in f if f.endswith(isdb_suffix)):
                                
             complete_file_path =r+'/'+file 
             dfisdb = pd.read_csv(complete_file_path)          
             'do something to the dfisdb'
             df = pd.merge (df, dfisdb)
  
 for r, d, f in os.walk(repository_path):
      for file in (f for f in f if f.endswith(sir_suffix)):
                                
             complete_file_path =r+'/'+file 
             dfsii = pd.read_csv(complete_file_path)          
             'do something to the dfsir'
             df = pd.merge (df, dfsir)
              
            prefix = 'treated_'
            df.to_csv(r+'/'+prefix+file, sep =',')

i would appreciate your ideas!

Luis

0 Answers
Related