Fully tokenize sentence including punctuation, contractions and hyphenated words

user_15303146 0
Viewed 67

I want to fully tokenize a sentence: "The element with the longest half-life is Uranium-234" said the professor.

I want this output:

['"', 'The', 'element', 'with', 'the', 'longests', 'half-life', 'isn't', 'Uranium-234', '"', 'said', 'the', 'professor', '.']

Here all the punctuation is separate, but the words like "isn't" and "doesn't" are one token. The hyphenated words are also seen as one token, which is what I want.

Currently I am using this to tokenize it:

p = re.compile(r"\w+(?:'\w+)?|[^\w\s]")
p.findall(s)

This gives me the output:

['"', 'The', 'element', 'with', 'the', 'longest', 'half', '-', 'life', 'isn't', 'Uranium', '-', '234', '"', 'said', 'the', 'professor', "."]

With this I can't tokenize the hyphenated words as one token.

python regex
1 Answers

Use a ['-] character class, and you forgot about an underscore:

\w+(?:['-]\w+)?|[^\w\s]|_

See proof.

EXPLANATION

--------------------------------------------------------------------------------
  \w+                      word characters (a-z, A-Z, 0-9, _) (1 or
                           more times (matching the most amount
                           possible))
--------------------------------------------------------------------------------
  (?:                      group, but do not capture (optional
                           (matching the most amount possible)):
--------------------------------------------------------------------------------
    ['-]                     any character of: ''', '-'
--------------------------------------------------------------------------------
    \w+                      word characters (a-z, A-Z, 0-9, _) (1 or
                             more times (matching the most amount
                             possible))
--------------------------------------------------------------------------------
  )?                       end of grouping
--------------------------------------------------------------------------------
 |                        OR
--------------------------------------------------------------------------------
  [^\w\s]                  any character except: word characters (a-
                           z, A-Z, 0-9, _), whitespace (\n, \r, \t,
                           \f, and " ")
--------------------------------------------------------------------------------
 |                        OR
--------------------------------------------------------------------------------
  _                        '_'

Python code:

import re
regex = r"\w+(?:['-]\w+)?|[^\w\s]|_"
test_str = "\"The element with the longest half-life is Uranium-234\" said the professor."
print(re.findall(regex, test_str))

Results: ['"', 'The', 'element', 'with', 'the', 'longest', 'half-life', 'is', 'Uranium-234', '"', 'said', 'the', 'professor', '.']

user_113298900
Related