We have a system that contains a data pipeline that computes reports, as well as an UI layer that can take ad-hoc queries, with some specific computation applied to our data set after query. I am thinking about migrating the existing batch data processing layer to Spark, but I am not so sure about the UI application layer.
The tricky part is that the request from UI also requires certain functions to be applied to the data set. I haven't found any query engine, such as Elasticsearch, that can apply custom function from query results. So I am wondering if it's a good idea to have UI layer trigger Spark directly, assuming the source data is already cached in Spark session.
So my questions are:
- Is Spark the right solution for real-time interactive applications? What are the alternatives?
- I would like to have the UI application runs outside of Spark cluster. So is there any framework that I can use for external service to trigger Spark job? Or leverage some existing Spark session?
Any advice would be greatly appreciated.