Converting Arabic Words to Unicode format in C#

user_5077196 0
Viewed 1738

I am designing an API where the API user needs Arabic text to be returned in Unicode format, to do so I tried the following:

public static class StringExtensions
{
    public static string ToUnicodeString(this string str)
    {
        StringBuilder sb = new StringBuilder();
        foreach (var c in str)
        {
            sb.Append("\\u" + ((int)c).ToString("X4"));
        }
        return sb.ToString();
    }
}

The issue with the above code that it returns the unicode of letters regardless of its position in word.

Example: let us assume we have the following word:

"سمير" which consists of:

'س' which is written like 'سـ' because it is the first letter in word.

'م' which is written like 'ـمـ' because it is in the middle of word.

'ي' which is written like 'ـيـ' because it is in the middle of word.

'ر' which is written like 'ـر' because it is last letter of word.

The above code returns unicode of { 'س', 'م' , 'ي' , 'ر'} which is:

\u0633\u0645\u064A\u0631

instead of { 'سـ' , 'ـمـ' , 'ـيـ' , 'ـر'} which is

\uFEB3\uFEE4\uFEF4\uFEAE

Any ideas on how to update code to get correct Unicode?

Helpful link

unicode c#
2 Answers

The string is just a sequence of Unicode code points; it does not know the rules of Arabic. You're getting out exactly the data you put in; if you want different data out, then put different data in!

Try this:

Console.WriteLine("\u0633\u0645\u064A\u0631");
Console.WriteLine("\u0633\u0645\u064A\u0631".ToUnicodeString());
Console.WriteLine("\uFEB3\uFEE4\uFEF4\uFEAE");
Console.WriteLine("\uFEB3\uFEE4\uFEF4\uFEAE".ToUnicodeString());

As expected the output is

سمير
\u0633\u0645\u064A\u0631
ﺳﻤﻴﺮ
\uFEB3\uFEE4\uFEF4\uFEAE

Those two sequences of Unicode code points render the same in the browser, but they're different sequences. If you want to write out the second sequence, then don't pass in the first sequence.

user_886560

Based on Eric's answer I knew how to solve my problem, I have created a solution on Github.

You will find a simple tool to run on Windows, and if you want to use the code in your projects then just copy paste UnicodesTable.cs and Unshaper.cs.

Basically you need a table of Unicodes for each Arabic letter then you can use something like the following extension method.

public static string GetUnShapedUnicode(this string original)
    {
        original = Regex.Unescape(original.Trim());
        var words = original.Split(' ');
        StringBuilder builder = new StringBuilder();
        var unicodesTable = UnicodesTable.GetArabicGliphes();
        foreach (var word in words)
        {
            string previous = null;
            for (int i = 0; i < word.Length; i++)
            {
                string shapedUnicode = @"\u" + ((int)word[i]).ToString("X4");
                if (!unicodesTable.ContainsKey(shapedUnicode))
                {
                    builder.Append(shapedUnicode);
                    previous = null;
                    continue;
                }
                else
                {
                    if (i == 0 || previous == null)
                    {
                        builder.Append(unicodesTable[shapedUnicode][1]);
                    }
                    else
                    {
                        if (i == word.Length - 1)
                        {
                            if (!string.IsNullOrEmpty(previous) && unicodesTable[previous][4] == "2")
                            {
                                builder.Append(unicodesTable[shapedUnicode][0]);
                            }
                            else
                                builder.Append(unicodesTable[shapedUnicode][3]);
                        }
                        else
                        {
                            bool previouChar = unicodesTable[previous][4] == "2";
                            if (previouChar)
                                builder.Append(unicodesTable[shapedUnicode][1]);
                            else
                                builder.Append(unicodesTable[shapedUnicode][2]);
                        }
                    }
                }

                previous = shapedUnicode;
            }
            if (words.ToList().IndexOf(word) != words.Length - 1)
                builder.Append(@"\u" + ((int)' ').ToString("X4"));
        }

        return builder.ToString();
    }
user_50771960
Related