Flink job .UnfulfillableSlotRequestException: Could not fulfill slot req. Req resource profile (ResourceProfile{UNKNOWN}) is unfulfillable

user_4632198 0
Viewed 7474

Flink job submission

$ ./bin/flink run -m 10.0.2.4:6123  /streaming/mvn-flinkstreaming-scala/mvn-flinkstreaming-scala-1.0.jar
Stream processing!!!!!!!!!!!!!!!!!
org.apache.flink.streaming.api.datastream.DataStreamSink@40ef3420

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
    ... 31 more
Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed
    ... 29 more
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not fulfill slot request 

    org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.

but when i checked the logs of the job in UI getting a different error,

Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
    ... 31 more
Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed
    ... 29 more
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException: Could not fulfill slot request ea

What should I check, my configuration parameter are as below, A) is the -m ip_address:6123 right option or 8081 should be the port... config..

# Note this accounts for all memory usage within the TaskManager process, including JVM metaspace and other overhead.

taskmanager.memory.process.size: 1568m

# To exclude JVM metaspace and overhead, please, use total Flink memory size instead of 'taskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.
#
# taskmanager.memory.flink.size: 1280m

# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.

taskmanager.numberOfTaskSlots: 2

# The parallelism used for programs that did not specify and other parallelism.

parallelism.default: 2

Cluster starting,

$  bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host centos1.
Starting taskexecutor daemon on host centos2.
Starting taskexecutor daemon on host centos3.

Able to process grep in master node,

]$ ps -ef | grep  flink
root      12300      1 10 07:22 pts/0    00:00:05 java -Xms16384m -Xmx16384m -Dlog.file=/storage/flink-1.10.0/log/

Unable to find a process of fink related to task managers, centos2 ~]$ psg flink

Is this a right state?

flink-streaming apache-flink
2 Answers

I've encountered this problem before, which is most likely an indicator for insufficient memory on your flink cluster. The different error messages also make sense, as they relate to each other.

Check your jobmanager.heap.size and taskmanager.heap.size in the config, increase them to a rather oversized amount and you should not see this error anymore. From here you can finetune the actual memory settings

user_60975600

I have faced the same problem in Flink-1.10.0. So please make sure you have enough memory according to the data load.

Error which i was getting: 

java.lang.OutOfMemoryError: Metaspace
  • jobmanager.heap.size: 1024m (Default)
  • taskmanager.memory.flink.size: 1280m (Default)
  • taskmanager.memory.jvm-metaspace.size: 256m (Default)

So I have increased taskmanager.memory.jvm-metaspace.size according to data load and it solved my issue.

For more details click here.

user_73178160
Related