I'm using a TStreamReader to read data from a file that purports to be utf-8. I have no problem reading the file until it comes to a section containing what appears to me to be a UTF-8 "£" symbol with the preceding xC2 missing - the file only contains the xA3 part of the character. I've traced this through the run-time library until it calls
Result := UnicodeFromLocaleChars(FCodePage, FMBToWCharFlags,
PAnsiChar(Bytes), ByteCount, nil, 0);
which returns 0 indicating that it doesn't like the input. Unfortunately the TStreamReader simply ends up discarding this buffer of input and then continues with the rest of the file without raising an error. This is extremely misleading about what the problem but that is just a side issue.
The issue appears to be a "defect" in the UTF-8 TEncoding class in that it simply discards the results of a failed conversion whilst the TStreamReader assumes that this isn't the behaviour of TEncoding.
I can work around this by using
Reader := TStreamReader.Create(FileStream, TMBCSEncoding.Create(CP_UTF8, 0, 0));
instead of
Reader := TStreamReader.Create(FileStream, TEncoding.UTF8);
as this makes it ignore the corrupt UTF-8 and simply include something (I haven't checked what) in my output. However, I would like to combine allowing the data through with reporting it and there doesn't seem to be any obvious way of doing this as the behaviour is hidden deep within the library.
Does anyone know of any standard Delphi library tools for doing this or do I need to resort to a lot of custom code?