Flink and k8s deployment

user_13617664 0
Viewed 597

Per Flink's doc, we can deploy a standalone Flink cluster on top of Kubernetes, using Flink’s standalone deployment， or deploy Flink on Kubernetes using native Kubernetes deployments.

The document says

We generally recommend new users to deploy Flink on Kubernetes using native Kubernetes deployments.

Is it because native Kubernetes is easier to get start with, or is it because standalone mode is kind of legacy ?

In native Kubernetes mode, Flink is able to dynamically allocate and de-allocate TaskManagers depending on the required resources. While in standalone mode, task managers have to be provisioned manually.

It sounds to me that native Kubernetes mode is a better choice.

apache-flink kubernetes
1 Answers

Posted community wiki based on other answers - David Anderson answer and austin_ce answer. Feel free to expand it.

Good explanation from the David Anderson answer:

Standalone mode:

In a Kubernetes session or per-job deployment, Flink has no idea it's running on Kubernetes. In this mode, Flink behaves as it does in any standalone deployment (where there is no cluster framework available to do resource management). Kubernetes just happens to be how the infrastructure was created, but as far as Flink is concerned, it could have been bare metal. You will have to arrange for kubernetes to create the infrastructure that you will have configured Flink to expect.

Native mode:

  • Session deployment

    In a Native Kubernetes session deployment, Flink uses its KubernetesResourceManager, which submits a description of the cluster it wants to the Kubernetes ApiServer, which creates it. As jobs come and go, and the requirements for task managers (and slots) go up and down, Flink is able to obtain and release resources from kubernetes as appropriate.

  • Application mode

    In Application Mode (blog post) (details) you end up with Flink running as a kubernetes application, which will automatically create and destroy cluster components as needed for the job(s) in one Flink application.

The native mode is recommended because it is just simpler, I would not say it is legacy:

The Native mode is the current recommendation for starting out on Kubernetes as it is the simplest option, like you noted. In Flink 1.13 (to be released in the coming weeks), there is added support for specifying Pod templates. One of the drawbacks to this approach is its limited ability to integrate with CI/CD.

user_163919910
Related