Is it possible to change process function code(behavior) in the running time?

Viewed 25

I just want to my flink application as much as configurable. And also i want to change the behavior of the process function in the running time instead of stopping the cluster and re-deploy the jar file.

Is there any document for that? Or is it possible to inject process function code into running jar. For instance, from the web ui, i will get the process function input(as a java code) then after submitting the form, I will update the process function behavior.

1 Answers

You can use a BroadcastProcessFunction (or a KeyedBroadcastProcessFunction), and on the broadcast channel, communicate (in some fashion) what the process function is supposed to do.

I've seen this technique used to broadcast javascript code (to be executed by Rhino), commands in a DSL, references to a JAR file to load, etc.

It's old and not well documented, but https://github.com/alpinegizmo/flink-training-exercises/blob/master/src/main/java/com/ververica/flinktraining/solutions/datastream_java/broadcast/TaxiQuerySolution.java is an example of this approach that uses Janino to compile and execute dynamically supplied Java expressions.

Related