I have a simple spark workflow.
- Task 1(spark batch job triggered job) creates a json object
- Task 2(spark streaming job that listens to a socket) uses this object and loads into database.
Task 1 data(json object) needs to be pushed to task 2 and Im wondering what would be the best way to do it?
Im planning to use Apache thrift server and streaming job connecting to this server.
Im wondering how can I push the data to thrift server.