Language detection in Python for big data

user_10763869 0
Viewed 1897

I am trying to run language detection on a Series object in a pandas dataframe. However, I am dealing with millions of rows of string data, and the standard Python language detection librarieslangdetect and langid are too slow, and after hours of running it still hasn't completed.

I set up my code as follows:

#function to detect language
def detect_language (cell):
    if len(cell) > 0:
        lan = langid.classify(cell)
    else:
        lan = "NaN"
    return lan

#language detection using langid module

df['language'] = df.apply(lambda row: detect_language(row.Series), axis = 1)

Does anybody have suggestions on how to speed up my code or if there is another library out there?

bigdata language-detection nlp python
3 Answers

You could use swifter to make your df.apply() more efficient. In addition to that, you might want to try whatthelang library which should be more efficient than langdetect.

user_45746330

Sample the data into multiple random sets and then use fasttext https://fasttext.cc to quickly get the results. It is fast and efficient.

import fasttext
path_to_pretrained_model = "lid.176.bin"
fmodel = fasttext.load_model(path_to_pretrained_model)
from datetime import datetime
df['language']=''
for i in df.index:
    ln_cnt = i
    result = fmodel.predict(str(df['prcsd_title'][i]))
    #print(result[0])
    #detector.FindLanguage(text=str(df['prcsd_title'][i]))
    df['language'][i]=result[0]
    if i%10000==0:
        now = datetime.now()
        current_time = now.strftime("%H:%M:%S")
        print("Current Time =", current_time)
        print(i)

You have to install Ubuntu if you are having Windows OS. Refer this video link https://www.youtube.com/watch?v=tQvghqdefTM&t=177s

user_118952790

Try:

from pandarallel import pandarallel
pandarallel.initialize(progress_bar=True)

"""
then instead of apply, use parallel_apply to use all the cores of your CPU to parallelize it
"""

df['new_column'] = df['your_column'].parallel_apply(lang_detect_func)
user_128819410
Related