Ideas to improve language detection between Spanish and Catalan

user_8461643 0
Viewed 510

I'm working on a text mining script in python. I need to detect the language of a natural language field from the dataset.

The thing is, 98% of the rows are in Spanish and Catalan. I tried using some algorithms like the stopwords one or the langdetect library, but these languages share a lot of words so they fail a lot.

I'm looking for some ideas to improve this algorithm.

One thought is, make a dictionary with some words that are specific to Spanish and Catalan, so if one text has any of these words, it's tagged as that language.

language-detection python
1 Answers

Approach 1: Distinguishing characters

Spanish and Catalan (note: there will be exceptions for proper names and loanwords e.g. Barça):

esp_chars = "ñÑáÁýÝ"
cat_chars = "çÇàÀèÈòÒ·ŀĿ"

Example:

sample_texts = ["El año que es abundante de poesía, suele serlo de hambre.",
                "Cal no abandonar mai ni la tasca ni l'esperança."]

for text in sample_texts:
    if any(char in text for char in esp_chars):
        print("Spanish: {}".format(text))
    elif any(char in text for char in cat_chars):
        print("Catalan: {}".format(text))

>>> Spanish: El año que es abundante de poesía, suele serlo de hambre.
    Catalan: Cal no abandonar mai ni la tasca ni l'esperança.

If this isn't sufficient, you could expand this logic to search for language exclusive digraphs, letter combinations, or words:

Spanish only Catalan only
Words como y su con él otro com i seva amb ell altre
Initial digraphs d' l'
Digraphs ss tj qü l·l l.l
Terminal digraphs ig

Catalan letter combinations that only marginally appear in Spanish

  • tx
  • tg          (Es. exceptions postgrado, postgraduado, postguerra)
  • ny          (Es. exceptions mostly prefixed in-, en-, con- + y-)
  • ll (terminal) (Es. exceptions (loanwords): detall, nomparell)

Approach 2: googletrans library

You could also use the googletrans library to detect the language:

from googletrans import Translator

translator = Translator()

for text in sample_texts:
    lang = translator.detect(text).lang
    print(lang, ":", text)

>>> es : El año que es abundante de poesía, suele serlo de hambre.
    ca : Cal no abandonar mai ni la tasca ni l'esperança.
user_90676150
Related