Iterating over two list of dataframes of different length and concatenating as dataframes them within a loop to perform a function

Question

Say, I have two lists of data frames. The size of the list of data frames are,

len(list_df1)
20
list_df1
[                 Gene_id  TCGA-A1-A0SM-01A-11R-A084-07  \
 0     ENSG00000000003_PC                  2.609759e+05   
 1     ENSG00000000005_PC                  2.046517e+02   
 2     ENSG00000000419_PC                  6.300425e+05   
 3     ENSG00000000457_PC                  7.922487e+04   
 4     ENSG00000000460_PC                  4.759842e+04   
 ...                  ...                           ...   
 999   ENSG00000067167_PC                  1.661041e+06   
 1000  ENSG00000067177_PC                  1.117150e+05   
 1001  ENSG00000067182_PC                  8.435913e+05   
 1002  ENSG00000067191_PC                  1.219213e+05   
 1003  ENSG00000067208_PC                  8.953860e+04   
 
       TCGA-A2-A04W-01A-31R-A115-07  TCGA-A2-A04X-01A-21R-A034-07  \
 0                     3.557029e+05                 112350.116669   
 1                     3.668742e+02                    678.138127   
 2                     9.395870e+05                 980843.179895   
 3                     7.663316e+04                  54672.074426   
 4                     3.543807e+04                  21895.818992   
 ...                            ...                           ...   
 999                   1.306383e+06                 652885.437844   
 1000                  1.706889e+05                  65106.955798   
 1001                  1.255371e+06                 634635.692737   
 1002                  4.121517e+04                  35705.970261   
 1003                  7.729493e+04                  51846.101598   
 
       TCGA-A2-A0D1-01A-11R-A034-07  TCGA-A2-A0EQ-01A-11R-A034-07  \
 0                     4.379606e+05                  2.218208e+05   
 1                     2.502301e+03                  1.585833e+03   
 2                     5.090120e+05                  6.800754e+05   
 3                     7.601743e+04                  6.207091e+04   
 4                     4.759907e+04                  4.022122e+04   
 ...                            ...                           ...

And for the second list of data frames,

len(list_df2)
15
list_df2
[                       Gene_id  TCGA-A1-A0SM-01A-11R-A084-07  \
 1580   ENSG00000082929_lncRNAs                    283.066402   
 1598   ENSG00000083622_lncRNAs                      0.000000   
 1995   ENSG00000093100_lncRNAs                    761.296884   
 2127   ENSG00000099869_lncRNAs                   4134.407284   
 2819   ENSG00000103472_lncRNAs                   4081.669994   
 ...                        ...                           ...   
 25447  ENSG00000224648_lncRNAs                      0.000000   
 25448  ENSG00000224649_lncRNAs                      0.000000   
 25450  ENSG00000224652_lncRNAs                  13698.679519   
 25452  ENSG00000224655_lncRNAs                    454.154778   
 25455  ENSG00000224658_lncRNAs                      0.000000   
 
        TCGA-A2-A04W-01A-31R-A115-07  TCGA-A2-A04X-01A-21R-A034-07  \
 1580                   16745.726795                      0.000000   
 1598                       0.000000                      0.000000   
 1995                    1091.806755                    504.529752   
 2127                    2058.792103                      0.000000   
 2819                    4541.654924                  12125.955565   
 ...                             ...                           ...   
 25447                      0.000000                      0.000000   
 25448                   1806.322490                      0.000000   
 25450                  20679.823343                   8560.813731   
 25452                      0.000000                      0.000000   
 25455                      0.000000                      0.000000   
 
        TCGA-A2-A0D1-01A-11R-A034-07  TCGA-A2-A0EQ-01A-11R-A034-07  \
 1580                   79172.350419                    365.577246   
 1598                       0.000000                      0.000000   
 1995                     698.135025                    294.962048   
 2127                    1053.164305                      0.000000   
 2819                    4517.449536                   9633.998928   
 ...                             ...                           ...   
 25447                      0.000000                      0.000000   
 25448                      0.000000                      0.000000   
 25450                   6887.145309                    931.142046   
 25452                    347.062760                    293.268032   
 25455                   1619.254228                      0.000000   
 
        TCGA-A2-A0T1-01A-21R-A084-07  TCGA-A2-A3XV-01A-21R-A239-07  \
 1580                    1024.779160                  29452.491771   
 1598                       0.000000                      0.000000   
 1995                     551.221323                    243.727431   
 2127                     415.769586                    551.508707   
 2819                   10530.579870                   2027.695619   
 ...                             ...                           ...

Now, what I need is, to concatenate the data frames from two lists into smaller data frames and run a function for each data frame.

Say my function is to perform correlation analysis,

import pandas as pd
from scipy.stats import pearsonr
    def correlation(df):
        correlations = pd.DataFrame()
    for PC in [column for column in df.columns if '_PC' in column]:
        for lncRNA in [column for column in df.columns if '_lncRNAs' in column]:
             correlations = correlations.append(pd.Series(pearsonr(df[PC],df[lncRNA]),index=['PCC', 'p-value'],name=PC + '_' +lncRNA))

And I need to use the above function for each concatenated data frames from the lists above. For example, this is what I tried.

for dfs,dfs2 in zip(list_df1,list_df2):
    #print(dfs.iloc[0:2,0:2],dfs2.iloc[0:2,0:2])
    df     = dfs
    df2    = dfs2
    DF = pd.concat([dfs,dfs2])
    
DF_T =DF.set_index("Gene_id").T

The DF_T looks like following,

  DF_T.shape()
(75, 1073)

However, DF_T is a concatenation of only last data frames from the 2 lists (list_df1,list_df2) which are defined above. Hence, if I run the correlation function on the data frame DF_T, it will only perform a correlation analysis only on the last data frames from the lists. Whereas, my aim is to run it all the small concatenated data frames from the two lists.

Therefore, I am looking for a way in which I can extract all data frames from two lists separately. Then, concatenate them into different data frames. Finally, apply the correlation function on each concatenated data frames. All these are done in order improve the performance speed of the correlation function. My input is multiple data frames of big size.

Any help or suggestions are much appreciated!!

user_4975981 · Answer

Using your own code, this should work:

def correlation(list_df1,list_df2):
    correlations = pd.DataFrame()
    for dfs,dfs2 in zip(list_df1,list_df2):
        DF = pd.concat([dfs,dfs2])
        df = DF.set_index("Gene_id").T
        for PC in [column for column in df.columns if '_PC' in column]:
            for lncRNA in [column for column in df.columns if '_lncRNAs' in column]:
                 correlations = correlations.append(pd.Series(pearsonr(df[PC],df[lncRNA]),index=['PCC', 'p-value'],name=PC + '_' +lncRNA))    

   return correlations

Iterating over two list of dataframes of different length and concatenating as dataframes them within a loop to perform a function

1 Answers