Issue with incremental encrpyption on specific Android device/OS

Expanding on Olivier's answer. He's determined that the exception is thrown by mMainDataStreamer.update(). If you look at AndroidKeyStoreCipherSpiBase class you will see that mMainDataStreamer is an instance of KeyStoreCryptoOperationChunkedStreamer class. Here's an interesting part:

// Binder buffer is about 1MB, but it's shared between all active transactions of the process.
// Thus, it's safer to use a much smaller upper bound.
private static final int DEFAULT_MAX_CHUNK_SIZE = 64 * 1024;

In our case, this default max chunk size is used. DEFAULT_MAX_CHUNK_SIZE puts an upper limit on the chunk size. If you pass bigger chunks to the cipher.update() method they will be cut into chunks of DEFAULT_MAX_CHUNK_SIZE. As you can see, even developers of Android didn't new the precise safe chunk size and had to guess themselves (unsuccessfully, in your case).

However, notice that Binder buffer is used to pass these chunks to the encryption process and results back from it. And it's size is only about 1MB.

Maybe there is an unusually small Binder buffer on this particular device? You can try to look into it by using that answer: https://stackoverflow.com/a/25666064/3249257

On future devices, you can use:

IBinder.getSuggestedMaxIpcSizeBytes()

https://developer.android.com/reference/android/os/IBinder#getSuggestedMaxIpcSizeBytes()

I've spent some time analyzing the issue. I couldn't find the real cause, but at least I have found some elements, so here they are.

Multi-part processing

First, you say that you use a chunk size of 32Kb, but it's not really true. You split the string in chunks of 32Kc (32768 characters), then you convert each chunk to a byte array. Since the UTF-8 representation of a char can range from 1 to 4 bytes, your byte array will generally be larger than 32Kb (unless you have only ASCII chars).

You should first convert the string to a byte array, then split it into 32Kb chunks. Only that will guarantee the size of the buffer that you pass to the crypto API.

Client-side error

Now, about the stacktrace you get. Contrary to what it seems at first glance, the error doesn't occur in doFinal() but in update(). When you call update(), the call is delegated to AndroidKeyStoreCipherSpiBase.engineUpdate(). The interesting part is:

try {
    flushAAD();
    output = mMainDataStreamer.update(input, inputOffset, inputLen);
} catch (KeyStoreException e) {
    mCachedException = e;
    return null;
}

It calls mMainDataStreamer.update(), which fails and throws a KeyStoreException with code KM_ERROR_MEMORY_ALLOCATION_FAILED. But the exception is caught, stored in mCachedException, and null is returned. That's why you get null when you call update().

When you call doFinal(), it calls AndroidKeyStoreCipherSpiBase.engineDoFinal():

protected final byte[] engineDoFinal(byte[] input, int inputOffset, int inputLen)
        throws IllegalBlockSizeException, BadPaddingException {
    if (mCachedException != null) {
        throw (IllegalBlockSizeException)
            new IllegalBlockSizeException().initCause(mCachedException);
    }

The method sees that there is a cached exception and throws it (wrapped in an IllegalBlockSizeException, which is totally unrelated to the real issue).

Keystore error

Now, the real issue. The actual work of encryption/decryption is performed by the Keystore service, which is a separate process written in C++. The relevant part for AES is in aes_operation.cpp.

There is a number of KM_ERROR_MEMORY_ALLOCATION_FAILED errors returned in that file. As its name suggests, the code means that a memory allocation has failed. So it seems that, for some reason, the Keystore was unable to allocate a buffer. It's hard to understand why.

Conclusion

Since the real cause is mysterious, I would suggest keeping a small buffer size and changing the splitting procedure as described at the beginning.