How to read an ANSI encoded file containing special characters

user_90745 0
Viewed 93780

I'm writing a TFS Checkin policy, which checks if our source files containing our file header.

My problem is, that our file header contains a special character "©" and unfortunately some of our source files are encoded in ANSI. So if I read these files in the policy, the string looks like this "Copyright � 2009".

string content = File.ReadAllText(pendingChange.LocalItem);

I tired to change the encoding of the string, but it does not help. So how can I read these files, that I get the correct string "Copyright © 2009"?

ansi encoding c#
3 Answers

I know this is an old question but I ran into a similar situation and found the accepted answer to be cutting some corners (no disregard for Jon Skeet's pragmatic short answer, but I'll flesh it out a little more)...

The specs state that the header will contain the encoding directly after {\rtf:

 \ansi  ANSI (the default)
 \mac   Apple Macintosh
 \pc    IBM PC code page 437 
 \pca   IBM PC code page 850, used by IBM Personal System/2 (not implemented in version 1 of Microsoft Word for OS/2)

According to Wikipedia the "ANSI character set has no well-defined meaning"

For the default ANSI you have the choice of these partially incompatible encodings:

using System.Text;
...
string content = File.ReadAllText(filename, Encoding.GetEncoding("ISO-8859-1"));
or
string content = File.ReadAllText(filename, Encoding.GetEncoding("Windows-1252"));

Using WordPad on windows 10 to save a file with a euro sign (0x80 in Windows-1252 but 0xA4 in ISO-8859-1) revealed the following:

The header stated the exact encoding after \ansi

{\rtf1\ansi\ansicpg1252\deff0\nouicompat\deflang1043{ ...

And the encoding was not directly used, instead it was wrapped in RTF encoding: \'80

according to the specs:

\'hh : A hexadecimal value, based on the specified character set (may be used to identify 8-bit values).

I guess the best thing to do is read the header, if the file starts with {\rtf1\ansi\ansicpg1252 then go for Windows-1252.

But to make things more complicated, the specs also state that there can be mixed encodings... search for '\upr'...

I guess there is no definitive answer, the easiest way to go in your case may be to search (in the un-decoded raw byte array) for all the variations of the encoded copyright signs that you may encounter in your source base.

In my case I finally decided to cut a few corners as well, but add a small percentage of defensive coding. All files I have seen so far were Windows-1252 so I common-case-optimised for that.

    Encoding encoding = Encoding.GetEncoding("Windows-1252", EncoderFallback.ReplacementFallback, DecoderFallback.ReplacementFallback);
    
    using (System.IO.StreamReader reader = new System.IO.StreamReader(filename, encoding)) {
        string header= reader.ReadLine();
        if (!header.Contains("cpg1252")) {
            if(header.Contains("\\pca"))
                encoding = Encoding.GetEncoding(850, EncoderFallback.ReplacementFallback, DecoderFallback.ReplacementFallback);
            else if (header.Contains("\\pc"))
                encoding = Encoding.GetEncoding(437, EncoderFallback.ReplacementFallback, DecoderFallback.ReplacementFallback);
            else
                encoding = Encoding.GetEncoding("ISO-8859-1", EncoderFallback.ReplacementFallback, DecoderFallback.ReplacementFallback);
        }
    }
    
    string content = System.IO.File.ReadAllText(filename, encoding);
user_6597780
Related