Detect language from string in PHP

Question

In PHP, is there a way to detect the language of a string? Suppose the string is in UTF-8 format.

user_1907888 · Answer

Text_LanguageDetect pear package produced terrible results: "luxury apartments downtown" is detected as Portuguese...

Google API is still the best solution, they give 300$ free credit and warn before charging you anything

Below is a super simple function that uses file_get_contents to download the lang detected by the API, so no need to download or install libraries etc.

function guess_lang($str) {

    $str = str_replace(" ", "%20", $str);

    $content = file_get_contents("https://translation.googleapis.com/language/translate/v2/detect?key=YOUR_API_KEY&q=".$str);

    $lang = (json_decode($content, true));

    if(isset($lang))
        return $lang["data"]["detections"][0][0]["language"];
 }

Execute:

echo guess_lang("luxury apartments downtown montreal"); // returns "en"

You can get your Google Translate API key here: https://console.cloud.google.com/apis/library/translate.googleapis.com/

This is a simple example for short phrases to get you going. For more complex applications you'll want to restrict your API key and use the library obviously.

user_3593449 · Answer

I have had good results with https://github.com/patrickschur/language-detection and am using it in production:

It uses ngrams in languages to detect the most likely language (the longer your string / the more words, the more accurate it will be), which seems like a solid proven method.
110 languages are supported, but you can also limit the number of languages to only those you are interested in.
Trainer and Language detector can easily be improved / customized. It uses the Universal Declaration of Human Rights in each of the languages as the foundation to detect a language, but if you know what type of sentences you experience you can easily extend or replace the used texts in each language and get better results fast. "Training" this library to become better is easy.
I would suggest to increase setMaxNgrams (I set it to 9000) in the Trainer and run it once, and then also use that setting in the Language detector class. Changing the ngrams number is a bit unintuitive (I had to look through the code to find out how it works), which is a drawback, and the default (310) is always too low in my opinion. More ngrams makes the guessing a lot better.
Because the library is very small, it was relatively easy to understand what is happening and how to tweak it.

My usage: I am analyzing emails for a CRM system to know what language an email was written in, so sending the text to a third party service was not an option. Even though the Universal Declaration of Human Rights is probably not the best basis to categorize the language of emails (as emails often have formulaic parts like greetings, which are not part of the Human Rights Declaration) it identifies the correct language in like 99% of cases, if there are at least 5 words in it.

Update: I managed to improve language recognition in emails to basically 100% when using the language-detection library with the following methods:

Add additional common phrases to the (relevant) language samples, like "Greetings", "Best regards", "Sincerely". These kind of expressions are not used in the Universal Declaration of Human Rights. Commonly used phrases help the language recognition a lot, especially formulaic ones used often my humans ("Hello", "Have a nice day") if you are analyzing human communication.
Set the maximum ngram length to 4 (instead of the default 3).
Keep the maxNgrams at 9000 as before.

These do make the library a bit slower, so I would suggest to use them in an async way if possible and measure the performance. In my case it is more than fast enough and much more accurate.

user_9588276 · Answer

I used this method to check for non- english, spanish, french chars using strictly PHP without any extra language API or Classes as of PHP 5.1. The language scripts list comes from: https://www.php.net/manual/en/regexp.reference.unicode.php See below

An improvement would be to add a function to PHP that lists all supported script languages so that you dont have to fill in the array by hand.

The usecase was for blocking non-latin posts to a form to improve it's spam blocking as the form was receiving a lot of russian, chinese, and arabic spam posts. Since this was implemented, its gone from 40000/week to less than 5, with none in the last 3 weeks. Google Re-Captcha was in use but it was being defeated easily. #satisfied

 $script)
   {
      if (!empty($script))
      {
         if (preg_match( '/[\p{'.$script.'}]/u', $text))
         {
            $found_scripts[] = $script;
         }
      }
   }

   return $found_scripts;
}

user_6792829 · Answer

Additional words for French and Spanish to Swiss Mister's answer:

    // Franch word list
    // from https://1000mostcommonwords.com/1000-most-common-french-words/
    $wordList['fr'] = array ('comme', 'que',  'était',  'pour',  'sur',  'sont',  'avec',
                             'être',  'à',  'un',  'ce',  'par',  'mais',  'que',  'est',
                             'il',  'eu',  'la', 'et', 'dans');

    // Spanish word list
    // from https://spanishforyourjob.com/commonwords/
    $wordList['es'] = array ('que', 'no', 'a', 'la', 'el', 'es', 'y',
                             'en', 'lo', 'un', 'por', 'qué', 'si', 'una',
                             'los', 'con', 'para', 'está', 'eso', 'las');

user_6350936 · Answer

My answer is for specific case. Here is what I wrote to find if string is in specific language, but there is one condition - different languages have different alphabets. In my case the word(s) can be in 3 languages - english, bulgarian and greek (each with different alphabet). And I need to find if a text is in bulgarian, so later translate it to greek.

class Language {
        protected $bgSymbols = array(
            'а', 'б', 'в', 'г', 'д', 'е', 'ж', 'з', 'и', 'й', 'к', 'л', 'м', 'н', 'о', 'п', 'р', 'с', 'т', 'у', 'ф', 'х', 'ц', 'ъ', 'ь', 'ч', 'щ', 'ш', 'ю', 'я',
            'А', 'Б', 'В', 'Г', 'Д', 'Е', 'Ж', 'З', 'И', 'Й', 'К', 'Л', 'М', 'Н', 'О', 'П', 'Р', 'С', 'Т', 'У', 'Ф', 'Х', 'Ц', 'Ъ', 'Ь', 'Ч', 'Щ', 'Ш', 'Ю', 'Я'
        );
        
        public function checkIfForTranslate($string) {
            $result = false;
            $stringArray = array();
            preg_match_all('/./u', $string, $matches);
            if(isset($matches[0])) {
                $stringArray = $matches[0];
            }
            foreach($this->bgSymbols as $symbol) {
                $found = array_search($symbol, $stringArray);
                if($found !== false) {
                    $result = true;
                    break;
                }
            }
            return $result;
        }
    }

Hope this help someone with similar case to mine.

Detect language from string in PHP

18 Answers