I build a distributed application with Django and server A needs to execute a task locally with thousands of arguments, asynchronously, and then use the task results in server B as the argument of a different task executed asynchronously. What is the best way to achieve that ?
Considerations
- There is no need to get the result of the remote task back in server A;
- Server A is located in different areas than server B;
- Performance is important.
One idea I wanted to explore is to POST the result of the local task to server B and execute the remote task asynchronously in sever B with apply_async(). But thousands of API calls that need to be autenticated will consume a lot of CPU resource I assume.
Would love to hear your thoughts, Thanks