I'm deploying Flink as application cluster mode. Deployment environment consists of two Kubernetes pod, one for the JobManager and one for the TaskManager (scalable). JobManager pod hosts one container based on a Docker image that execute the command:
standalone-job.sh start-foreground --job-classname=${JOB_CLASS_NAME}
The deployment process is based on a CI-CD pipeline and works in this way.
- Write the code
- Git commmit and push
- Jenkins pipeline starts JAR isbuilded
- A new pod is created and the JAR is deployed on it
I cannot stop the previous application using savepoint, so I have to retain checkpoints after cancellation and try to use the last retained checkpoint to restore state.
The problem is that after code updates, the pipeline spreads a new pod with a new JAR with a completely new job with a new jobId. The command to restore from checkpoint seems to be
standalone-job.sh start-foreground --job-classname=${JOB_CLASS_NAME} --fromSavepoint /path/to/checkpoint
but /path/to/checkpoint is not fixed because contains the previous jobId that I cannot access.
Is there an automation to restore state without passing the checkpoint path?