I'm in the process of evaluation Apache Flink for potential use case and I'm struggling how I should model my computations in Flink itself.
In my case I will have many (unknown upfront) small, heterogeneous processing graphs, each of them could use parts of standard Flink DataStream API to process data from external sensors. Each of theses graphs won't be computationally expensive. My first thought was to make each of those small graphs separate job and deploy it to Flink cluster. The problem is that because task slots are not shared between subtasks from different jobs I'm facing a situation where I would need to create task managers with very high number of task slots but everywhere I read it is recommended to have number of slots equal to number of cores in the system.
I've found an article about using Flink when the nature of the job is somehow dynamic (https://techblog.king.com/rbea-scalable-real-time-analytics-king/) but implementing some kind of custom DSL on top of Flink in a situation where most of the graphs that will be created can be easily expressed using DataStream does not look like an elegant solution to me.
Is Flink just not designed to handle dynamic number of jobs that are defined using DataStream API and the Flink way would be to model my use case with single meta-job that will be generic enough to be able to express all potential computations?