I'm trying to run a beam pipeline using Flink as the runner deployed using Native Kubernetes application mode but I'm unable to configure Beam on how to use the Flink interface in application mode.
Since Flink is deployed in application mode, there isn't a master manager node up and running to receive new jobs. Specifically, the docs for Beam to use the Flink runner on a separate cluster needs a flink-master URL which I don't have.
Start JobService with Flink Rest endpoint:
docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081
To run Flink jobs in native k8s application mode, a command like this is executed:
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=my.cluster.id \
-Dkubernetes.container.image=custom-docker-image \
-Dkubernetes.rest-service.exposed.type=ClusterIP \
-Dkubernetes.pod-template-file.jobmanager=/path/to/jobmanager/template \
-Dkubernetes.pod-template-file.taskmanager=/path/to/taskmanager/template \
local:///opt/flink/usrlib/job.jar
Is there a way to have Beam not use the flink-master url but instead build the complete job jar and submit it to Kubernetes as a Flink application in the form of such a command?