How to remove interpunct and number which followed by fullstop by using regular expressions

Admin 1
Viewed 218

I have a begginer in NLP and I have a dataset for NLP task which has strings. I want to clean it by removing interpunct and nuber which followed by fullstop, such as 'George is · working here since 2015.' to 'George is working here since'.

I want to use regular expression and I think comile library works in my problem. The code that I have is

def stripPunc(wordList):
    """Strips punctuation from list of words"""
    puncList = ["]","[","·",".",";",":","!","?","/","\\",",","#","@","$","&",")","(","\""]
    for punc in puncList:
        for word in wordList:
            wordList=[word.replace(punc,'') for word in wordList]
    return wordList

but returns

['G',
 'e',
 'o',
 'r',
 'g',
 'e',
 ' ',
 'i',
 's',
 ' ',
 '',
 ' ',
 '',
 '',
 ' ',
 'w',
 'o',
 'r',
 'k',
 'i',
 'n',
 'g',
 ' ',
 'h',
 'e',
 'r',
 'e',
 ' ',
 's',
 'i',
 'n',
 'c',
 'e',
 ' ',
 '2',
 '0',
 '1',
 '5',
 '']

instead of 'George is working here since'.

One more aproach is to use

import re

re_word_pattern = re.compile(r'\w+')
re_brackets = re.compile(r'(\[|\])')
re_number_to_zero = re.compile(r'\d+')
re_interpunct_to_zero = re.compile(r'')
text = 'George is · [] working here since 2015.'
text = re_brackets.sub('', text)
text = re_number_to_zero.sub('', text)

which gives

George is ·  working here since .

so in this case how could I remove the interpunct?

nlp python regex
2 Answers

You can use

def clean_text(text):
    return " ".join(re.sub(r'\s*\d+\.|[^\w\s]|_', '', text).split())

See a Python demo:

import re
s='George is · working here since 2015.'
print( " ".join(re.sub(r'\s*\d+\.|[^\w\s]|_', '', s).split()) )
# => George is working here since

Details:

  • re.sub(r'\s*\d+\.|[^\w\s]|_', '', s) - removes a couple of patterns:
    • \s*\d+\. - zero or more whitespaces, one or more digits and a dot
    • | - or
    • [^\w\s] - any punctuation other than _
    • |_ - or a _
  • " ".join(...).split() shrinks the whitespace in the result.
user_38329700

In your first code, you are iterating through a list, so if you give it a string it will iterate through each character so what you can do is :

def stripPunc(sentence):
    """Strips punctuation from list of words"""
    puncList = ["]","[","·",".",";",":","!","?","/","\\",",","#","@","$","&",")","(","\""]
    for punc in puncList:
        sentence = sentence.replace(punc,'')
    return sentence
user_160859010
Related