Detecting hashed words and phrases to obscure them

user_484337 0
Viewed 69

We have a list of "severe" words which we want to match and obscure against incoming sentences.

e.g.

words = ['shoots', 'forest floor', 'leaves']
sentence = 'The panda eats, shoots, and leaves on the forest floor.'
obscure(words, sentence)
=> 'The panda eats, s*****, and l***** on the f********.'

The difficult bit is that some of the severe words are so severe that we store them in a hashed format. So our annotators may input shoots but it is stored as 92415d1bb58ff22c1a98cb610bf0d774. Then;

words = ['92415d1bb58ff22c1a98cb610bf0d774', 'forest floor', 'leaves']
sentence = 'The panda eats, shoots, and leaves on the forest floor.'
obscure(words, sentence)
=> 'The panda eats, s*****, and l***** on the f********.'

For single words (shoots) we can easily chop up the input sentence and match even the hashed words.

But multi-words (forest floor) is challenging. I have yet to figure out an elegant regex or iterative method for doing this.

Ideally we solve this in Python though JavaScript and Ruby are available to us if they have existing solutions or a large benefit.

python regex string
2 Answers

You can reduce your input string by iterating over the words list until the string is empty, running replacements when matches are found:

from hashlib import md5
words = ['shoots', 'forest floor', 'leaves']
sentence = 'The panda eats, shoots, and leaves on the forest floor.'
def obscure(s, w):
   d = ''
   while s:
      if (k:=[i for i in w if s.startswith(i)]):
         d += (m:=max(k, key=len))[0]+('*'*(len(m) - 1))
         s = s[len(m):]
      elif (j:=[t for i in range(len(s)) if md5((t:=s[:i]).encode()).hexdigest() in w]):
         d += (m:=max(j, key=len))[0]+('*'*(len(m) - 1))
         s = s[len(m):]
      else:
         d += s[0]
         s = s[1:]
   return d

print(obscure(sentence, words))

Output:

The panda eats, s*****, and l***** on the f***********

This will also work with hashes:

words = ['92415d1bb58ff22c1a98cb610bf0d774', 'forest floor', 'leaves']
sentence = 'The panda eats, shoots, and leaves on the forest floor.'
print(obscure(sentence, words))

Output:

The panda eats, s*****, and l***** on the f***********.
user_73267380

For not having the actual string in your code, you can still have a regex to match it, here is an example :

import hashlib
import re

blacklist = [('92415d1bb58ff22c1a98cb610bf0d774', re.compile('\bs\w{5}\b'), "s")]
#            the hash                             a matching regex       the first letter


def censor(text: str) -> str:
    for hashed_value, regex, starting_letter in blacklist:
        for match in reversed(tuple(regex.finditer(text))):
            if hashlib.md5(match.group(0).encode("utf-8")).hexdigest() == hashed_value:
                text = text[:match.start()] + starting_letter + ("*" * (len(match.group()) - 1)) + text[match.end():]
    return text


if __name__ == '__main__':
    print(censor('The panda shines, eats, shoots, and leaves on the forest floor. '
                 'He has fashootsies from Shoots city.'))
    # The panda shines, eats, s***** and leaves on the forest floor. He has fas*****es from Shoots city.
user_113841840
Related