I have a list of words and a list of sentences.
In a loop, I am trying to identify whether words occur in a varying subset of sentences.
In each iteration the subset of sentences changes.
I want to make sure that the words only exist as the exact words and not as compounds or parts e.g. I want to only identify "test" and not "testcenter".
Which lead me to a simple regex pattern: rf"\b{word}\b"
I distribute the task and what follows to multiple processes using joblib (worth because lists very large and function after is heavy).
The problem is that the operation is quite slow. My idea to speed up the process was to pre-compile this pattern for all existing words and store them in a dictionary to look them up instead of recompiling at each step.\
To my surprise this approach actually takes longer than compiling at each step (roughly 28x). Can somebody please explain why, since as far as I understand Python dictonary look-ups should be quite swift and regex compilation is expensive.
The process looks like this:
from joblib import Parallel, delayed
import random as rn
import re
def check_occurence_and_x(word, sentences, regex_patterns):
try:
reg_pat = regex_patterns[word]
except:
reg_pat = re.compile(rf"(\b{word}\b)", re.IGNORECASE)
found = False
if not (reg_pattern.search(sentences) == None):
found = True
if found:
"do more"
cores = 6
reg_pattern_dic = {}
for word in all_words:
reg_pattern_dic[word] = re.compile(rf"\b{word}\b", re.IGNORECASE)
for iteration in range(x):
sampled_sentences = rn.sample(all_sentences, 10PERCENT)
Parallel(n_jobs=cores, backend="multiprocessing", verbose=0, timeout=None)(delayed(check_occurence_and_x)(word, sampled_sentences, reg_pattern_dic) for word in all_words)