Determine a string's encoding in C#

It depends where the string 'came from'. A .NET string is Unicode (UTF-16). The only way it could be different if you, say, read the data from a database into a byte array.

This CodeProject article might be of interest: Detect Encoding for in- and outgoing text

Jon Skeet's Strings in C# and .NET is an excellent explanation of .NET strings.

My finally working approach is to try potential candidates of expected encodings by detecting invalid characters in the strings created from the byte array by the encodings. If I don't encounter invalid characters, I suppose the tested encoding works fine for the tested data.

For me, having only Latin and German special characters to consider, in order to determine the proper encoding for a byte array, I try to detect invalid characters in a string with this method:

    /// <summary>
    /// detect invalid characters in string, use to detect improper encoding
    /// </summary>
    /// <param name="s"></param>
    /// <returns></returns>
    public static bool DetectInvalidChars(string s)
    {
        const string specialChars = "\r\n\t .,;:-_!\"'?()[]{}&%$§=*+~#@|<>äöüÄÖÜß/\\^€";
        return s.Any(ch => !(
            specialChars.Contains(ch) ||
            (ch >= '0' && ch <= '9') ||
            (ch >= 'a' && ch <= 'z') ||
            (ch >= 'A' && ch <= 'Z')));
    }

(NB: if you have other Latin-based languages to consider, you might want to adapt the specialChars const string in the code)

Then I use it like this (I only expect UTF-8 or Default encoding):

        // determine encoding by detecting invalid characters in string
        var invoiceXmlText = Encoding.UTF8.GetString(invoiceXmlBytes); // try utf-8 first
        if (StringFuncs.DetectInvalidChars(invoiceXmlText))
            invoiceXmlText = Encoding.Default.GetString(invoiceXmlBytes); // fallback to default

I found new library on GitHub: CharsetDetector/UTF-unknown

Charset detector build in C# - .NET Core 2-3, .NET standard 1-2 & .NET 4+

it's also a port of the Mozilla Universal Charset Detector based on other repositories.

CharsetDetector/UTF-unknown have a class named CharsetDetector.

CharsetDetector contains some static encoding detect methods:

• CharsetDetector.DetectFromFile()
• CharsetDetector.DetectFromStream()
• CharsetDetector.DetectFromBytes()

detected result is in class DetectionResult has attribute Detected which is instance of class DetectionDetail with below attributes:

• EncodingName
• Encoding
• Confidence

below is an example to show usage:

// Program.cs
using System;
using System.Text;
using UtfUnknown;

namespace ConsoleExample
{
    public class Program
    {
        public static void Main(string[] args)
        {
            string filename = @"E:\new-file.txt";
            DetectDemo(filename);
        }

        /// <summary>
        /// Command line example: detect the encoding of the given file.
        /// </summary>
        /// <param name="filename">a filename</param>
        public static void DetectDemo(string filename)
        {
            // Detect from File
            DetectionResult result = CharsetDetector.DetectFromFile(filename);
            // Get the best Detection
            DetectionDetail resultDetected = result.Detected;

            // detected result may be null.
            if (resultDetected != null)
            {
                // Get the alias of the found encoding
                string encodingName = resultDetected.EncodingName;
                // Get the System.Text.Encoding of the found encoding (can be null if not available)
                Encoding encoding = resultDetected.Encoding;
                // Get the confidence of the found encoding (between 0 and 1)
                float confidence = resultDetected.Confidence;

                if (encoding != null)
                {
                    Console.WriteLine($"Detection completed: {filename}");
                    Console.WriteLine($"EncodingWebName: {encoding.WebName}{Environment.NewLine}Confidence: {confidence}");
                }
                else
                {
                    Console.WriteLine($"Detection completed: {filename}");
                    Console.WriteLine($"(Encoding is null){Environment.NewLine}EncodingName: {encodingName}{Environment.NewLine}Confidence: {confidence}");
                }
            }
            else
            {
                Console.WriteLine($"Detection failed: {filename}");
            }
        }
    }
}

example result screenshot:

As others have mentioned, a string in C# is always encoded as UTF-16LE (System.Text.Encoding.Unicode).

Reading between the lines, what I believe what you're actually concerned about is whether or not the characters in your string are compatible with some other known encoding (i.e. will they "fit" in that other code page?).

In that case, the most correct solution I've found is to attempt the conversion and see if the string changes. If a character in your string doesn't "fit" in the destination encoding, the encoder will substitute it for some sentinel character that will (e.g. '?' is common).

    // using System.Text;

    // And if you're using the "System.Text.Encoding.CodePages" NuGet package, you 
    // need to call this once or GetEncoding will raise a NotSupportedException:
    // Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);

    var srcEnc = Encoding.Unicode;
    var dstEnc = Encoding.GetEncoding(1252); // 1252 Requires use of the "System.Text.Encoding.CodePages" NuGet package.
    string srcText = "Some text you want to check";
    string dstText = dstEnc.GetString(Encoding.Convert(srcEnc, dstEnc, srcEnc.GetBytes(srcText)));

    // if (srcText == dstText) the srcText "fits" (it's compatible).
    // else the srcText doesn't "fit" (it's not compatible)