Specifying keytab for FlinkDeployment of Flink Kubernetes Operator

Question

I run Flink application in Kubernetes with the help of Flink Kubernetes Operator. This application authenticates in Kafka broker via Kerberos. Therefore I need to inject a keytab file into the application.

My FlinkDeployment descriptor contains a PodTemplate, which mounts the keytab file from a Secret to /opt/keytab. The PodTemplate is at the spec level, hence effective for both JobManager and TaskManager.

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
...
spec:
  image: ...
  flinkVersion: v1_15
  flinkConfiguration:
    security.kerberos.krb5-conf.path: /etc/krb5.conf
    security.kerberos.login.keytab: /opt/keytab
    security.kerberos.login.principal: ...
    security.kerberos.login.contexts: KafkaClient
  ...
  podTemplate:
    spec:
      containers:
      - name: flink-main-container
        volumeMounts:
        - name: kerberos-keytab
          mountPath: /opt/keytab
          subPath: keytab
...

When submitting this FlinkDeployment to the Kubernetes cluster, the Flink application never starts. In the logs of the flink-kubernetes-operator Pod I see error messages about missing file: /opt/keytab.

Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not create Kubernetes cluster "flink-app-test".
    at org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:292)
    at org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:211)
    at org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)
    at org.apache.flink.kubernetes.operator.service.FlinkService.submitApplicationCluster(FlinkService.java:200)
    at org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:155)
    at org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:52)
    at org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractJobReconciler.restoreJob(AbstractJobReconciler.java:188)
    at org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractJobReconciler.reconcileSpecChange(AbstractJobReconciler.java:122)
    at org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:145)
    at org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:55)
    at org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:115)
    ... 13 more
Caused by: org.apache.flink.configuration.IllegalConfigurationException: Kerberos login configuration is invalid: keytab [/opt/keytab] doesn't exist!
    at org.apache.flink.runtime.security.SecurityConfiguration.validate(SecurityConfiguration.java:151)
    at org.apache.flink.runtime.security.SecurityConfiguration.(SecurityConfiguration.java:97)
    at org.apache.flink.runtime.security.SecurityConfiguration.(SecurityConfiguration.java:69)
    at org.apache.flink.kubernetes.kubeclient.decorators.KerberosMountDecorator.(KerberosMountDecorator.java:59)
    at org.apache.flink.kubernetes.kubeclient.factory.KubernetesJobManagerFactory.buildKubernetesJobManagerSpecification(KubernetesJobManagerFactory.java:64)
    at org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployClusterInternal(KubernetesClusterDescriptor.java:274)
    ... 23 more

This behaviour seems quite surprising. I expected the keytab file to be used in JobManager Pod and in TaskManager Pods, but not in flink-kubernetes-operator Pod. The idea to mount the keytab file to the flink-kubernetes-operator Pod looks wrong. Different applications may run with different keytabs.

What is the proper way to provide keytab (and also krb5.conf) files to Flink applications when running them with Flink Kubernetes Operator?

UPDATE 1

It turns out that Flink Kubernetes Operator automatically creates volume mounts with krb5.conf and keytab - for JobManager as well as TaskManager Pods. This occurs when we specify the following settings for a FlinkDeployment:

  flinkConfiguration:
    security.kerberos.krb5-conf.path: /path/to/krb5.conf
    security.kerberos.login.keytab: /path/to/keytab

This behavior is implemented in KerberosMountDecorator.

Therefore both krb5.conf and keytab should be mounted to the flink-kubernetes-operator Pod. This means, that we have to customize the Helm chart of Flink Kubernetes Operator by configuring the necessary volumes. Edit values.yaml:

operatorVolumeMounts:
  create: true
  data:
  - name: kerberos-krb5conf
    mountPath: /path/to/krb5.conf
    subPath: krb5.conf
  - name: kerberos-keytab
    mountPath: /path/to/keytab
    subPath: keytab

operatorVolumes:
  create: true
  data:
  - name: kerberos-krb5conf
    configMap:
      name: kerberos-krb5conf
  - name: kerberos-keytab
    secret:
      secretName: kerberos-keytab

Practically this means, that

we may have only one keytab per Kubernetes Namespace
we have to customize the Helm chart of Flink Kubernetes Operator for each Namespace - specify the keytab specific to this Namespace

This is not so convenient. It would be more convenient to configure Kerberos authentication at the application level, rather than at the Kubernetes Namespace level.

Specifying keytab for FlinkDeployment of Flink Kubernetes Operator

0 Answers