I have to implement a document management service as an orchestration layer that orchestrates between underlying service such as storage , parsing , anti virus scan etc. The requirement is to make the layer flexible so that different flows for different kinds of documents can be implemented quickly
One approach is to model this as a event driven system and implement processing pipeline on events using framework like Apache Flink.
Another way to think about it is - workflow . Design this as a workflow that runs on a workflow engine like Apache Airflow or Uber Cadence.
What will be a better approach.
Are workflow engine and event stream framework alternatives for an orchestration layer?
Viewed 1394
2 Answers
Disclaimer: I'm a tech lead of Temporal Workflow and former tech lead of Cadence Workflow. But I know something about Flink and Airflow :).
It would be pretty hard to implement orchestration using Flink. The reason is that Flink is a stream processing solution that is optimized for very fast processing of each request. Orchestration should deal with slow requests or services being down for a long time. Flink doesn't support such scenarios out of the box. It also models processing as a static graph while orchestration is frequently a very dynamic state machine.
Airflow is also based on a static graph. So it is not really suitable for complex scenarios that do not fit into its DAG paradigm. The even more severe problem with Airflow is its very limited scalability. That's why I've never heard it being used for service orchestration outside of the data pipeline control plane where scalability is not really needed.
Temporal Workflow was built from the beginning as a very dynamic and highly scalable orchestration system. There are many high throughput service orchestration use cases in production at many companies that rely on Temporal. Some of them are listed in the case studies.
There are folks building platforms on top of Flink that support use cases more like what you are describing than what is straightforwardly done with Flink itself. In particular, I want to suggest you take a look at Stateful Functions, which is a new open source library from Ververica, the original creators of Flink. From what I've understood of your requirements, this seems like it might be a good fit.
Cogynt is another example of this trend of building platforms on top of Flink to support use cases that have been difficult to implement with Flink itself, while leveraging its high-performance, fault tolerant, exactly-once stream processing engine.
Disclaimer: I work for Ververica.
Related