Flink job failed with "Checkpoint Coordinator is suspending."

user_14131268 0
Viewed 2031

I ran a flink job and it failed after 18 hours. Failure message: Checkpoint Coordinator is suspending.

Checkpoint screenshot: Checkpoint screenshot

Job overview screenshot: Job overview screenshot

Here is the job manager logs:

2020-10-10 13:53:10,636 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: kafkaSource -> Timestamps/Watermarks -> Process (1/1) (c38419ece8208c1ef2948087f2b84dd0) switched from RUNNING to FAILED.
java.lang.Exception: Could not perform checkpoint 3265 for operator Source: kafkaSource -> Timestamps/Watermarks -> Process (1/1).
    at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:785)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$triggerCheckpointAsync$3(StreamTask.java:760)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.run(StreamTaskActionExecutor.java:87)
    at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:261)
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:485)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:469)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:708)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:533)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1394)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:974)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:870)
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:843)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:776)
    ... 11 more

I don't know what cause this problem.

flink-streaming apache-flink
1 Answers

This can happen when your application is trying to checkpoint, and at that time the checkpoint coordinator (Job Manager) shuts down due to some reason, and the checkpoint could not be completed. The reason for the shutdown can be due to multiple reasons, for example, you started a new deployment, you canceled the job, the job had to exit due to some runtime exception, etc.

user_65918370
Related