Remove rows in which string contains other letters than A,C,T,G,N

user_14914490 0
Viewed 236

I'm fairly new to numpy and pandas, let's say that I have a 2D numpy array and I need to delete all rows in which the second value contain only the letters 'A', 'C', 'T', 'G' and 'N'

file = 
[['id' 'genome'],
 ['0' 'ATGTTTGTTTTT'],
 ['1' 'ATGTTTGTXXXX'],
 ['2' 'ATGDD2GTTTTT']
]

so after filtering I can get this

[['id' 'genome'],
 ['0' 'ATGTTTGTTTTT']]

I wanted to do 3 for loops that are checking each char one by one but this is sooo slow when I have 500 rows

pandas numpy arrays python
3 Answers

Use Series.str.contains with values and ^ for start and $ for end of string:

file = [['id', 'genome'],
 ['0', 'ATGTTTGTTTTT'],
 ['1', 'ATGTTTGTXXXX'],
 ['2', 'ATGDD2GTTTTT']
]
 
df = pd.DataFrame(file[1:], columns=file[0])
print (df)


df = df[df['genome'].str.contains('^[ACTGN]+$')]
print (df)
  id        genome
0  0  ATGTTTGTTTTT
user_29010020

One other option is str.match with the same pattern as in @jezrael's answer:

df = df[df['genome'].str.match('^[ACTGN]+$')]

Also, we can detect the illegal character with negation on str.contains:

# [^ACTGN] detects any characters that are not ACTGN
df = df[~df['genome'].str.contains('[^ACTGN]')]
user_42384080

The other answers are probably more efficient as they use native pandas functionality. If you're more handy with Python then you can use map to perform the filtering.

df = DataFrame(
    data={
        'id': [0, 1, 2],
        'genome': ['ATGTTTGTTTTT', 'ATGTTTGTXXXX', 'ATGDD2GTTTTT']
    }
)

filtered_df = df[df['genome'].map(lambda x: re.sub('[ACTGN]+', '', x) == '')]

In this last line:

  • Use map to run a function on each row
  • The lambda syntax means: for each value x use the function re.sub('[ACTGN]+', '', x) == ''
  • The re.sub function says replace any character in ACTGN with blank.
  • After replacing the known characters with '', if the string is empty then we want to keep the row.

The advantage of this approach is that you can use any python code on the pandas values.

filtered_df = df[df['genome'].map(lambda x: <any-code-you-want-that-yields-a-boolean>)]
user_62525250
Related