Flink Queryable State Out Of Memory Error

user_6073494 0
Viewed 64

I am trying to use queryable state on Flink (version 1.14.0) but unfortunately, I keep getting the following error:

2021-11-07 11:10:55,795 ERROR org.apache.flink.queryablestate.network.AbstractServerHandler [] - Error while handling request with ID 1
java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
    at org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:632) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:607) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:202) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:172) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocate(PoolArena.java:134) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocate(PoolArena.java:126) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:395) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.network.NettyBufferPool.ioBuffer(NettyBufferPool.java:95) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.network.messages.MessageSerializer.writePayload(MessageSerializer.java:203) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.network.messages.MessageSerializer.serializeRequest(MessageSerializer.java:96) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.network.ServerConnection$EstablishedConnection.sendRequest(ServerConnection.java:403) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.network.ServerConnection$EstablishedConnection.sendRequest(ServerConnection.java:284) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.network.ServerConnection.sendRequest(ServerConnection.java:91) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.network.Client.sendRequest(Client.java:168) ~[flink-dist_2.12-1.14.0.jar:1.14.0]
    at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.lambda$getState$2(KvStateClientProxyHandler.java:187) ~[flink-queryable-state-runtime-1.14.0.jar:1.14.0]
    at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) ~[?:?]
    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:829) [?:?]
flink-streaming apache-flink
1 Answers

The error itself is quite clear. Basically, you are out of memory as per the java.lang.OutOfMemoryError being shown in the stack trace. It's likely that you either didn't allocate enough memory to the job or it's exceeded the amount.

Broadly speaking, queryable state isn't designed to be used in any type of production capacity (it's better suited for situations like debugging, etc.), so you may want to consider if it's the best fit for your use-case as well.

user_5574450
Related