Pandas str.contains() not working in some cases

Question

I have a script, that does linear modelling between pairs of conditions: The dataframe looks like this:

   Accession                                   Sequence variable        value
0     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    39.300171
1     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   132.637125
2     O14548                          [R].gLPDQMLYr.[T]     DMSO  1165.245826
3     O14548                          [R].gLPDQMLYr.[T]     DMSO   642.971908
4     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    83.906058
5     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   160.718841
6     O14548                          [R].gLPDQMLYr.[T]     DMSO  1240.856710
7     O14548                          [R].gLPDQMLYr.[T]     DMSO   557.508092
8     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    56.228425
9     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   302.346775
10    O14548                          [R].gLPDQMLYr.[T]     DMSO  1176.998098
11    O14548                          [R].gLPDQMLYr.[T]     DMSO   766.993819
12    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.387985
13    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.175678
14    O14548                          [R].gLPDQMLYr.[T]     CCCP   885.174420
15    O14548                          [R].gLPDQMLYr.[T]     CCCP   130.458963
16    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.557088
17    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.095801
18    O14548                          [R].gLPDQMLYr.[T]     CCCP   612.171540
19    O14548                          [R].gLPDQMLYr.[T]     CCCP    46.449990
20    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     6.016590
21    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.466220
22    O14548                          [R].gLPDQMLYr.[T]     CCCP   586.392482
23    O14548                          [R].gLPDQMLYr.[T]     CCCP   303.857624
24    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]      C+I    44.627773
25    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]      C+I     0.841494
26    O14548                          [R].gLPDQMLYr.[T]      C+I   632.355914
27    O14548                          [R].gLPDQMLYr.[T]      C+I   162.333292
28    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]      C+I    12.075158
29    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]      C+I   154.253098
30    O14548                          [R].gLPDQMLYr.[T]      C+I   159.767999
31    O14548                          [R].gLPDQMLYr.[T]      C+I  1031.399087
32    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]      C+I   150.724386
33    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]      C+I   260.684163
34    O14548                          [R].gLPDQMLYr.[T]      C+I   141.459156
35    O14548                          [R].gLPDQMLYr.[T]      C+I   262.659208

I now want to fit a linear model for each pair. I get the pairs by the following code:

def tessa(source):
    result = []
    for p1 in range(len(source)):
            for p2 in range(p1+1,len(source)):
                    result.append([source[p1],source[p2]])
    return result

unique_conditions = list(set(conditions))
pairs = tessa(unique_conditions)
print(pairs)

I am looping over the pairs and filtering by dataframe for the conditions:

for pair in pairs:
        
        pair.sort()
        print(pair)
        print(pair[0],pair[1])
        temp=melted_Peptides[(melted_Peptides['variable'].str.contains(pair[0]))|(melted_Peptides['variable'].str.contains(pair[1]))]
        print(temp)

Here comes the problem. It does not filter correctly .The output of this:

['C+I', 'CCCP']
C+I CCCP
   Accession                                   Sequence variable       value
12    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP    0.387985
13    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP    0.175678
14    O14548                          [R].gLPDQMLYr.[T]     CCCP  885.174420
15    O14548                          [R].gLPDQMLYr.[T]     CCCP  130.458963
16    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP    0.557088
17    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP    0.095801
18    O14548                          [R].gLPDQMLYr.[T]     CCCP  612.171540
19    O14548                          [R].gLPDQMLYr.[T]     CCCP   46.449990
20    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP    6.016590
21    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP    0.466220
22    O14548                          [R].gLPDQMLYr.[T]     CCCP  586.392482
23    O14548                          [R].gLPDQMLYr.[T]     CCCP  303.857624

While for the next comparison it looks okay:

['CCCP', 'DMSO']
CCCP DMSO
   Accession                                   Sequence variable        value
0     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    39.300171
1     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   132.637125
2     O14548                          [R].gLPDQMLYr.[T]     DMSO  1165.245826
3     O14548                          [R].gLPDQMLYr.[T]     DMSO   642.971908
4     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    83.906058
5     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   160.718841
6     O14548                          [R].gLPDQMLYr.[T]     DMSO  1240.856710
7     O14548                          [R].gLPDQMLYr.[T]     DMSO   557.508092
8     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    56.228425
9     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   302.346775
10    O14548                          [R].gLPDQMLYr.[T]     DMSO  1176.998098
11    O14548                          [R].gLPDQMLYr.[T]     DMSO   766.993819
12    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.387985
13    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.175678
14    O14548                          [R].gLPDQMLYr.[T]     CCCP   885.174420
15    O14548                          [R].gLPDQMLYr.[T]     CCCP   130.458963
16    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.557088
17    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.095801
18    O14548                          [R].gLPDQMLYr.[T]     CCCP   612.171540
19    O14548                          [R].gLPDQMLYr.[T]     CCCP    46.449990
20    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     6.016590
21    O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     CCCP     0.466220
22    O14548                          [R].gLPDQMLYr.[T]     CCCP   586.392482
23    O14548                          [R].gLPDQMLYr.[T]     CCCP   303.857624

For the third it looks weird again:

['C+I', 'DMSO']
['C+I', 'DMSO']
C+I DMSO
   Accession                                   Sequence variable        value
0     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    39.300171
1     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   132.637125
2     O14548                          [R].gLPDQMLYr.[T]     DMSO  1165.245826
3     O14548                          [R].gLPDQMLYr.[T]     DMSO   642.971908
4     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    83.906058
5     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   160.718841
6     O14548                          [R].gLPDQMLYr.[T]     DMSO  1240.856710
7     O14548                          [R].gLPDQMLYr.[T]     DMSO   557.508092
8     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO    56.228425
9     O14548  [K].lAGAWASEAYSPQGLkPVVSTEAPPIIFATPTk.[L]     DMSO   302.346775
10    O14548                          [R].gLPDQMLYr.[T]     DMSO  1176.998098
11    O14548                          [R].gLPDQMLYr.[T]     DMSO   766.993819

I am using the same code for approx. 5000 different dataframes and it always works. The conditions are named exactly the same, but somehow it breaks in some cases.

Can anybody please help?

user_2901002 · Answer

You can add regex=False parameter for avoid convert values to regex in Series.str.contains:

melted_Peptides['variable'].str.contains(pair[0], regex=False)

Pandas str.contains() not working in some cases

1 Answers