I'm running into a problem whereby my distributed cluster appears to "hang" - e.g. tasks stop processing and hence a backlog of unprocessed tasks builds up so I'm looking for some way to help debug what's going on.
On the Client there's the processing method which will tell me what tasks are currently running on each worker but AFAICS that's the only info about the tasks available on the Client object?
What I'd like to to is to be able to query not just processing tasks, but all tasks including processed, processing and errored and for each task to be able to get some statistics such as submitted_time and completion_time which would allow me to find out what tasks are blocking the cluster.
This would be similar to the extended metadata on the ipyparallel.AsyncResult
A nice to have would be to to be able to get the args/kwargs for any give task too. This would be especially helpful in debugging failed tasks.
Is any of this functionality available currently or is there any way to get the info I'm after?
Any other suggestions on how to debug the problem would be greatly welcomed.