Error which "shouldn't happen" caused by MalformedInputException when reading file to string with UTF-16

This is a bug introduced in JDK 17.

Prior to this version, this Error throwing code was only used for the String constructor which indeed can never encounter a CharacterCodingException because it configures the decoder to substitute illegal content.

E.g., when you use

String s = new String(new byte[] { 50 }, StandardCharsets.UTF_16);
System.out.println(s.chars()
    .mapToObj(c -> String.format(" U+%04x", c)).collect(Collectors.joining("", s, "")));

you’ll get

� U+fffd

In JDK 17, the code has been refactored and code duplication removed. Now, the same method decodeWithDecoder will be used for both, the String constructor and Files.readString. But Files.readString is supposed to report encoding errors instead of substituting the problematic content. Therefore, the decoder has not been configured to substitute malformed content, intentionally.

When you run

Path p = Files.write(Files.createTempFile("charset", "test"), new byte[] { 50 });
try(Closeable c = () -> Files.delete(p)) {
    String s = Files.readString(p, StandardCharsets.UTF_16);
}

under JDK 16, you’ll correctly get

Exception in thread "main" java.nio.charset.MalformedInputException: Input length = 1
        at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
        at java.base/java.lang.StringCoding.newStringNoRepl1(StringCoding.java:1053)
        at java.base/java.lang.StringCoding.newStringNoRepl(StringCoding.java:1003)
        at java.base/java.lang.System$2.newStringNoRepl(System.java:2265)
        at java.base/java.nio.file.Files.readString(Files.java:3353)
        at first.test17.CharsetProblem.main(CharsetProblem.java:23)

The now-removed dedicated routine threw the MalformedInputException encapsulated in an IllegalArgumentException. The immediate caller looks like

/*
 * Throws CCE, instead of replacing, if unmappable.
 */
static byte[] getBytesNoRepl(String s, Charset cs) throws CharacterCodingException {
    try {
        return getBytesNoRepl1(s, cs);
    } catch (IllegalArgumentException e) {
        //getBytesNoRepl1 throws IAE with UnmappableCharacterException or CCE as the cause
        Throwable cause = e.getCause();
        if (cause instanceof UnmappableCharacterException) {
            throw (UnmappableCharacterException)cause;
        }
        throw (CharacterCodingException)cause;
    }
}

and there lies the problem. When the code was refactored to use the same routine for the String constructor and Files.readString, this caller was not adapted. It still expects an IllegalArgumentException where the common method now throws an Error. Or the common method should have been adapted to better suit both cases, e.g. by having a parameter telling whether CharacterCodingException exceptions should be possible or not.

It’s worth noting that the charset decoding code has a lot of optimizations and shortcuts for commonly used charsets. That’s why you rarely get to this specific method. UTF-16 seems to be one (if not the) rare case where this method is used.

Different things going on here. But, yeah, it sure looks like you found a JVM bug! congratulations, I think :)

But, some context to explain precisely what's going on and what you found. I think your code's got bigger problems of your own making, and once you solve those, the JVM bug will no longer be a problem for you (but, by all means, do report it!). I'll try to cover all concerns:

1. Your code is broken because UTF-8 and UTF-16 are fundamentally incompatible. The upshot is that saving an even amount of characters as UTF-8 is likely to result in something that can be read with UTF-16 without error, although what you read will be utter gobbledygook. With an odd number of characters, you'll run into decoding errors.

2. The JVM is buggy! You found a JVM Bug - the effect of the decoding error should not be than an Error is thrown. The specific bug is that substitution doesn't actually cover all failure conditions, but the code is written with the assumption that it would.

3. The bug appears to be related to improper application of lenient mode, which requires explaining what substitution and underflow is.

UTF-8 vs. UTF-16

• When you convert characters to bytes or vice versa, you are using a charset encoding.
• Files are byte sequences, not characters.
• There are no exceptions to these rules.

Hence, if you are typing characters, and saving, and you're not picking a charset encoding? Somebody is. If you're bashing on your keyboard in notepad.exe and saving, then notepad's picking one for you. You can't not have an encoding.

To try to explain the nuances of what happens here, forget about programming for a moment.

We decide on a protocol: You think of a way to describe a person using a single adjective; you write it down on a piece of paper (just the adjective) and give it to me. I then read it and guess which of our circle of friends you are attempting to describe. I happen to be bilingual, and speak fluent dutch and english. You don't know this, or you do but we never discussed this part of the protocol between us two.

You begin, and think of a particularly lanky person, so you decide to write down "slim", on the note. You leave the room, I enter, and I pick up the note.

I make a wrong assumption and I assume you wrote it in dutch instead, so I read this note, and, thinking you wrote it in dutch, I read 'slim', which is an actual dutch word, but it means "smart". Had you written down, say, "tall" on your note instead, this would not have occurred: "Tall" is not in the dutch dictionary, hence I'd know that you made an 'error' (you wrote an invalid word. It was valid to you, but I'm reading it assuming its dutch, so I'd think you made a mistake). But, "slim", those 4 exact letters, so happens to be both valid dutch AND valid english, but it doesn't mean the same thing at all.

UTF-8 vs UTF-16 is exactly like that: There are character sequences you can encode with UTF-16 that produce a byte stream, which so happens to also be entirely valid UTF-8, but it means something completely different, and vice versa! But there are also sequences of characters that, if saved as UTF-16 and then read as UTF-8 (or vice versa) would be invalid.

So, the "slim" situation can occur, and the "tall" situation can occur. Either one is mostly useless to you: When I read your note and see "Slim", and I thought that meant 'smart', we still 'lost' and I picked the wrong friend - no better a result. So what point is there, right? Anytime you convert chars to bytes and back again, every conversion step along the path needs to use the exact same encoding for all that beforehand or its never going to work.

But HOW it fails - that's the rub: When you wrote "slim" - I just picked the wrong friend. When you wrote "tall", I exclaimed that an error had occurred as that isn't a dutch word.

UTF-16 translates each character into a sequence of 2, 3, or 4 bytes depending on the character. When you save plain jane ascii characters as UTF-8, they all end up being 1 byte, and in general any 2 such bytes, decoded as a single UTF-16 character, 'is valid' (but a completely different character, completely unrelated to the input!), so if you save 8 ASCII chars as UTF-8 (or ASCII - boils down to the same stream of bytes), and then read it as UTF-16, it's highly likely to not throw any exceptions. You get a 4-length string of gobbledygook out, though.

Let's try it!

String test = "gerikg";
byte[] saveAsUtf8 = test.getBytes(StandardCharsets.UTF_8);
String readAsUtf16 = new String(saveAsUtf8, StandardCharsets.UTF_16);
System.out.println(test);
System.out.println(readAsUtf16);

... results in:

gerikg
来物歧

See? Complete gobbledygook - unrelated chinese characters came out.

But, now lets go with an odd number:

String test = "gerikgw";
byte[] saveAsUtf8 = test.getBytes(StandardCharsets.UTF_8);
String readAsUtf16 = new String(saveAsUtf8, StandardCharsets.UTF_16);
System.out.println(test);
System.out.println(readAsUtf16);

gerikgw
来物歧�

Note that weird question mark thing: That's a glyph (a glyph is an entry in a font: The symbol used as representing some character) that indicates: Something went wrong here - this isn't a real character, but an error in decoding.

But, shove gerikgw in a text file (make sure it has no trailing enter, as that's a symbol too), and run your code, and indeed - JVM BUG! Nice find!

Substitution

That weird question mark symbol thing is a 'substitution'. UTF encoders can encode any 32-bit value. The unicode system has 32-bits worth of addressable characters (actually, not quite, it's less, some slots are intentionally marked as not used and will never be, for fun reasons but too unrelated to go into), but not every single one of them available is 'filled'. There's room for new characters if we need em for later. Also, not every sequence of bytes is neccessarily valid UTF-8.

So, what to do when 'invalid' input is detected? One option, in strict parsing mode, is to crash (throw something). Another is to 'read' the error as the 'error' character (shown with that question mark glyph when you print it to a screen) and pick up where we left off. UTF is a pretty cool formatting system that 'knows' when a new character starts, thus, you can never get an offset issue (where we're 'offset by half' and keep reading stuff wrong because of misalignment).

The JVM bug

This explains the code you've pasted: That malformed encoding stuff 'cannot occur', as per the comment, because lenient mode is on, so any errors should just result in substitutions. Except it is right there, this is a really dumb error, one of those that really result in the author of this code visibly and audibly slapping their forehead in pure shame:

In this case, there's a single remaining byte in the sequence of bytes left, but in UTF-16 world, all valid byte representations are at least 2 bytes. This condition is called underflow and the decoder (CharsetDecoder cd) isn't buggy - it correctly detects this situation, thus, if (!cr.isUnderflow()) cr.throwException(); results in cr.throwException() being executed, which - naturally, throws MalformedInputException, which is a subtype of CharacterCodingException, thus, code hops straight to the catch 4 lines below which then says "This cannot happen".

Conclusion, author had a brainfart moment. Only 2 things can be true:

1. underflows cannot ever occur here, ever. The brainfart is that there's an if in there that checks for the impossible, that is pointless.
2. underflows CAN occur here, and the comment in the catch block is therefore incorrect. Substitution doesn't fix this problem.

The correct code would presumably be, instead:

private static int decodeWithDecoder(CharsetDecoder cd, char[] dst, byte[] src, int offset, int length) {
    ByteBuffer bb = ByteBuffer.wrap(src, offset, length);
    CharBuffer cb = CharBuffer.wrap(dst, 0, dst.length);
    try {
        CoderResult cr = cd.decode(bb, cb, true);
        if (!cr.isUnderflow())
            cr.throwException();
        cr = cd.flush(cb);
        if (!cr.isUnderflow()) cb.write(SUBSTITUTION_CHAR);
    } catch (CharacterCodingException x) {
        // Substitution is always enabled,
        // so this shouldn't happen
        throw new Error(x);
    }
    return cb.position();
}

In other words - if underflow occurs, emit one subtitution char (to represent the 'un-character' represented by that dangling single byte that doesn't mean anything), and just return the result. After all, that fits the strategy of lenient mode, and the comment says that we're evidently in lenient mode ("Substitution is enabled").

I suggest you file a bug at the open JDK project, or search for this first.

To work around it until its fixed...

The workaround

Replace:

Files.readString(file, StandardCharsets.UTF_16);

with:

fixedReadString(file, StandardCharsets.UTF_16);

...

public static String fixedReadString(Path file, Charset charset) {
  try {
   Files.readString(file, StandardCharsets.UTF_16);
  } catch (Error e) {
    if (!(e.getCause() instanceof MalformedInputException)) throw e;
    // see notes
  }
}

The one remaining question is what you want to do when this occurs. The input is definitely problematic, I generally despise 'lenient' mode. So I'd just throw new MalformedInputException and in general rewrite it all to use strict mode instead. However, if you want to duplicate the intended effect (which is: "来物歧�" - which isn't useful, but it is what the code was supposed to return), that's not actually all that easy to recreate. You can pray that just adding a random character at the end (say, a space) and re-parsing will hopefully at least produce something, you could rewrite the entire functionality of Files.readString itself (not too complicated), or just return "�"; - tossing away the entire string and just leaving that one substitution character, which should at least help someone debug into: Ah, right, I'm using the wrong charset to read this file.