On Dask, calling sparse iterative methods Dask Array or across nodes in a cluster

user_3166083 0
Viewed 88

I have a large sparse matrix that can only fit on multiple nodes of a cluster. I would like to solve a system of equations by calling something like 

import scipy.sparse.linalg as splinalg

A = my_sparse_matrix          # a sparse dask array defined across four nodes
b = my_vector                 # an initial vector
M = my_sparse_preconditioner  # a preconditioner to help improve convergence of algorithm

x = splinalg.bicgstab(A, b, M=M)

Is there a way to do this so that

  • It will solve for x in parallel if I am only using one node?
  • Will it work if I am using more than one node (4 nodes for example)?

If not, is this something that the Dask community is working on?

dask-distributed dask linear-algebra parallel-processing
1 Answers

Dask array supports sparse matrix blocks.

Dask arrays support the SciPy LinearOperator API

These two facts may be enough to solve your problem, but you will still undoubtedly need to do some work yourself. For example there is not obvious storage solution for distributed sparse arrays. There is no obvious way to partition data, etc..

So "yes", Dask is able to help here, but "no" there is no obvious canned solution for you.

user_6166160
Related