How to build a Python wheel with compiled Fortran extension module without requiring a specific MinGW version on the user's system?

user_8028981 0
Viewed 2072

As far as I understand, one of the main advantages of distributing Python packages through wheels is that I can include extension modules in a compiled form. Then, the user of the package is not required to have a system that allows compilation of the source code.

Now I managed to build a wheel for my package that includes a Fortran extension module. The computer on which I built has Windows7 64, and Python 3.6.

In order to get everything running, I followed this very helpful guideline (many thanks to Michael Hirsch). One of the steps was to install MinGW-64 with the following settings: Architecture: x86_64, Threads: posix, Exception: seh.

I then installed the Python package on another testing machine (Win10 64, Python 3.6) from that wheel:

D:\dist2>pip install SMUTHI-0.2.0a0-cp36-cp36m-win_amd64.whl
Processing d:\dist2\smuthi-0.2.0a0-cp36-cp36m-win_amd64.whl
Requirement already satisfied: scipy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: sympy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: argparse in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: numpy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: matplotlib in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: pyyaml in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: six>=1.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: python-dateutil in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pytz in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: cycler>=0.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Installing collected packages: SMUTHI
Successfully installed SMUTHI-0.2.0a0

However, when I started a test run of the program, I encountered the following error:

D:\dist2>smuthi example_input.dat
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\smuthi.exe\__main__.py", line 5, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\__main__.py", line 4, in <module>
    import smuthi.read_input
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\read_input.py", line 3, in <module>
    import smuthi.simulation
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\simulation.py", line 8, in <module>
    import smuthi.t_matrix as tmt
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\t_matrix.py", line 6, in <module>
    import smuthi.nfmds.t_matrix_axsym as nftaxs
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\nfmds\t_matrix_axsym.py", line 11, in <module>
    import smuthi.nfmds.taxsym
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

The extension .pyd file (taxsym.cp36-win_amd64.pyd) was at its place - just Python couldn't load it.

Next, I uninstalled MinGW from the testing machine and reinstalled MinGW-64 with the same settings that I had used on the building machine (see above). Afterwards, I could run the program, and Python was able to correctly load the extension module.

My question is: Does anybody have an idea why the error occurred in the first place? And how can I avoid that the user of my Python package has to have a specific version of MinGW installed (or even any) for the package to work properly?

Edit: A small example that reproduces the error:

Minimal example

File structure:

setup.py
example/
    __init__.py
    run_hello.py
    extension_package/
        __init__.py             
        fortran_hello.f90

The setup.py reads:

import setuptools
from numpy.distutils.core import Extension
from numpy.distutils.core import setup

setup(
   name="example",
   version="0.1",
   author="My Name",
   author_email="my@email.com",
   description="Example package to demonstrate wheel issue",
   packages=['example', 'example.extension_package'],
   ext_modules=[Extension('example.extension_package.fortran_hello',
                          ['example/extension_package/fortran_hello.f90'])],
)

The run_hello.py reads:

import example.extension_package.fortran_hello
example.extension_package.fortran_hello.hello()

The fortran_hello.f90 reads:

subroutine hello
print *,"Hello World!"
end subroutine hello

Creation of the wheel

I ran python setup.py bdist_wheel which resulted in the file example-0.1-cp36-cp36m-win_amd64.whl

Installation of the package on machine with correct MinGW version

D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1

D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
 Hello World!
>>> exit()

This is as it should be.

Installation of the package on machine without correct MinGW version

To reproduce the error, I renamed the MinGW folder on the testing machine to some other name and then:

D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1

D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\site-packages\example\run_hello.py", line 1, in <module>
    import example.extension_package.fortran_hello
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.
python-wheel mingw fortran dll python
2 Answers

I can confirm that the steps described in rmar_'s answer create a pyd file that can be imported from a machine that has no MinGW installed. This is really helpful!

In order to create a binary wheel, I do the following:

  1. Follow the steps described in rmar_'s answer in order to generate statically linked pyd files in place.
  2. Run python setup.py bdist_wheel afterwards.

For the second step, I switch off the compilation of the pyd file (e.g., by removing the ext_modules keyword from the setup() call), because I want that the pyd file created in the first step is used (and not a newly created, which might be not statically linked).

I don't know if this makes sense at all, but it seems to work ...

user_80289810
Related