Fatal Python error when using a dynamic version of Python to execute embedded python code

Viewed 1223

SPOILER: partially solved (see at the end).

Here is an example of code using Python embedded:

#include <Python.h>
int main(int argc, char** argv)
{
    Py_SetPythonHome(argv[1]);
    Py_Initialize();
    PyRun_SimpleString("print \"Hello !\"");
    Py_Finalize();
    return 0;
}

I work under Linux openSUSE 42.2 with gcc 4.8.5 (but I also have the same problem on openSUSE 13.2 with gcc 4.8.3 or RedHat 6.4 with gcc 4.4.7).

I compiled a static and a dynamic version of Python 2.7.9 (but I also have the same problem with Python 2.7.13).

I compile my example linking to the static version of Python with the following command:

g++ hello.cpp -o hello \
-I /home/caduchon/softs/python/2.7.9/64/gcc/4.8.5/static/include/python2.7 \
-L /home/caduchon/softs/python/2.7.9/64/gcc/4.8.5/static/lib \
-l python2.7 -l pthread -l util -l dl

If I execute my example with the static version of Python in argument, it works.

If I execute it on the dynamic version of Python in argument, I get the following error (it happens in Py_Initialize()):

> ./hello /home/caduchon/softs/python/2.7.9/64/gcc/4.8.5/dynamic
Fatal Python error: PyThreadState_Get: no current thread
Aborted (core dumped)

I have no idea why it works with static version and it doesn't with the dynamic one. How can I solve this kind of problem ?

EDIT: my script installing Python is the following:

#!/bin/bash

WORKDIR=/home/caduchon/tmp/install_python_2_7_13
ARCHIVEDIR=/home/caduchon/downloads/python
PYTHON_VERSION='2.7.13'
EZ_SETUP_VERSION='0.9'
SETUPTOOLS_VERSION='34.1.0'
CYTHON_VERSION='0.25.2'
NUMPY_VERSION='1.12.0'
SCIPY_VERSION='0.18.1'
MATPLOTLIB_VERSION='2.0.0'
INSTALLDIR=/home/caduchon/softs/python/$PYTHON_VERSION/64/gcc/4.8.5
LAPACKDIR=/home/caduchon/softs/lapack/3.6.1/64/gcc/4.8.5

### Tkinter ###
echo "Install Tkinter"
sudo apt-get install tk-dev

### Workdir ###
echo "Create workdir"
mkdir -p $WORKDIR/static
mkdir -p $WORKDIR/dynamic

### Python
for x in static dynamic
do
    echo "Install Python ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/Python-$PYTHON_VERSION.tgz .
    tar -xzf ./Python-$PYTHON_VERSION.tgz &> archive.log
    cd ./Python-$PYTHON_VERSION
    echo "  configure"
    if [ "$x" = "static" ]
    then
        ./configure --prefix=$INSTALLDIR/$x --libdir=$INSTALLDIR/$x/lib &> configure.log
    else
        export LD_RUN_PATH=$INSTALLDIR/$x/lib
        ./configure --enable-shared --prefix=$INSTALLDIR/$x --exec-prefix=$INSTALLDIR/$x --libdir=$INSTALLDIR/$x/lib &> configure.log
    fi
    echo "  build"
    make &> make.log
    echo "  install"
    make install &> make_install.log
    echo "  done"
done

### setuptools
for x in static dynamic
do
    echo "Install setuptools ($x)"
    cd $WORKDIR/$x
    echo "  extract archives"
    cp $ARCHIVEDIR/ez_setup-$EZ_SETUP_VERSION.tar.gz .
    tar -xzf ./ez_setup-$EZ_SETUP_VERSION.tar.gz &> archive.log
    cp $ARCHIVEDIR/setuptools-$SETUPTOOLS_VERSION.zip .
    unzip ./setuptools-$SETUPTOOLS_VERSION.zip &> archive.log
    cp ./ez_setup-$EZ_SETUP_VERSION/ez_setup.py ./setuptools-$SETUPTOOLS_VERSION/.
    cd ./setuptools-$SETUPTOOLS_VERSION
    echo "  install"
    $INSTALLDIR/$x/bin/python ./ez_setup.py &> setup.log
    echo "  done"
done

### Cython
for x in static dynamic
do
    echo "Install Cython ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/Cython-$CYTHON_VERSION.tar.gz .
    tar -xzf ./Cython-$CYTHON_VERSION.tar.gz &> archive.log
    cd ./Cython-$CYTHON_VERSION
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

### NumPy
for x in static dynamic
do
    echo "Install NumPy ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/numpy-$NUMPY_VERSION.zip .
    unzip ./numpy-$NUMPY_VERSION.zip &> archive.log
    cd ./numpy-$NUMPY_VERSION
    echo "  build"
    $INSTALLDIR/$x/bin/python ./setup.py build --fcompiler=gfortran &> build.log
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

### SciPy
for x in static dynamic
do
    echo "Install SciPy ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/scipy-$SCIPY_VERSION.tar.gz .
    tar -xzf ./scipy-$SCIPY_VERSION.tar.gz &> archive.log
    cd ./scipy-$SCIPY_VERSION
    echo "  configure"
    echo "[DEFAULT]" > ./site.cfg
    echo "library_dirs = $LAPACKDIR/lib64" >> ./site.cfg
    echo "search_static_first = true" >> ./site.cfg
    echo "  build"
    $INSTALLDIR/$x/bin/python ./setup.py build --fcompiler=gfortran &> build.log
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

### MatPlotLib
for x in static dynamic
do
    echo "Install MatPlotLib ($x)"
    cd $WORKDIR/$x
    echo "  extract archive"
    cp $ARCHIVEDIR/matplotlib-$MATPLOTLIB_VERSION.tar.gz .
    tar -xzf ./matplotlib-$MATPLOTLIB_VERSION.tar.gz &> archive.log
    cd ./matplotlib-$MATPLOTLIB_VERSION
    echo "  build"
    $INSTALLDIR/$x/bin/python ./setup.py build &> build.log
    echo "  install"
    $INSTALLDIR/$x/bin/python ./setup.py install &> install.log
    echo "  done"
done

EDIT: I identified a possible cause of the problem. If I remove the line export LD_RUN_PATH=$INSTALLDIR/$x/lib in the installation of dynamic Python, my embedded code works. I printed sys.path through embedded code and it point to the right installation. BUT... in this way I can't use the installation directly : it loads a wrong version found in the system (when I print sys.path I see it points to /usr/...). Also, I don't want to have to set environment variables to launch Python because I use several versions of Python on the same machine.

EDIT: Keeping my default installation script of Python, I solve the problem by adding -rdynamic in the options when compiling the C++ example. But I don't well understand what is this option, and which kind of disaster it can cause...

1 Answers
Related