Chapel-Python integration questions

user_2268791 0
Viewed 226

I'm trying to see if I can use Chapel for writing parallel code for use in a Python-based climate model: https://github.com/CliMT/climt

I don't have any experience with Chapel, but it seems very promising for my use-case. I had a few questions about how to integrate Chapel code into my current workflow:

  1. I know you can build importable .so files, but can the compilation stop when the Cython file is generated? I can then include it into the distribution and use standard setuptools to compile my project on Travis.

  2. Can I pass numpy arrays to a Python extension written in Chapel?

  3. If answer to 2. is yes, and my computation is embarassingly parallel in one dimension of the array, is there an elegant way to express this paralellism in Chapel?

  4. If I write Chapel code that works on multiple nodes and compile it to a Python extension, how do I run it? Can I use mpirun python my_code.py kind of a command?

parallelism-amdahl chapel parallel-processing python
1 Answers

  1. Unfortunately not currently. However, we do leave the generated .pxd and .py(x) files in the directory with the .so, so you could make use of those in the meanwhile (this wasn't a feature request we've considered, so if you felt motivated, definitely feel free to open an issue on our Github page: https://github.com/chapel-lang/chapel/issues).

    For reference, we do this because the Cython compilation command is rather tricky. I had thought we printed the Cython command used with the chpl compilation flag --print-commands, but that doesn't look to be the case (I'll make an issue for that).

  2. You can pass 1 dimensional numpy arrays of known primitive types to Chapel from Python. We're hoping to add support for other numpy arrays soon (hopefully in 1.21, slated for March 2020)

  3. This is definitely doable on arrays in Chapel - I would recommend using a forall loop when traversing this dimension of your array for your computation, which will divide the indices in that dimension into a number of tasks determined by Chapel. (For those not familiar with forall loops, this link gives a good overview of the concept)

For example:

forall x in arr.domain.dim(1) {
  // traverses the first dimension of arr's domain in parallel
  ...
}
  1. If you compile your Chapel library into a Python extension with multilocale settings, you can specify the number of locales (nodes) needed using the numlocales argument to the extension's chpl_setup function. Doing so will take care of distributing the Chapel code for you when you run your Python program.

For example, you could write:

import MyChplLib

MyChplLib.chpl_setup(4)
...

to run your program with 4 locales (nodes).

I should probably mention that as of the 1.20 release, we don't have support for array arguments in multilocale libraries. We're still figuring out priorities for the 1.21 release, so feedback on how fast you want that would be super helpful!

user_74908870
Related