How to compile python3 on RHEL with SSL? SSL cannot be imported

Question

I'm trying to compile python on RHEL because my current python is using an old 1.0.2k ssl version.

(test_env) [brad@reason tlscheck]$ python3 --version
Python 3.9.3
(test_env) [brad@reason tlscheck]$ python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.0.2k-fips  26 Jan 2017
(test_env) [brad@reason tlscheck]$ openssl version
OpenSSL 1.1.1l  24 Aug 2021

I think the issue is that when I compiled 3.9.3, I had not updated my OpenSSL version. I have since updated my OpenSSL and need to use it with python. So I have downloaded the newest python 3.10, but in the make stage I get an error that it will not make with ssl. I the following message:

Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl                                     


Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer

This is the full log of trying to compile: https://pastebin.com/36EntpFz

When I use the configure options that @tony-yip mentioned, I get the following in my configure.

checking for openssl/ssl.h in /etc/ssl... no
checking whether compiling and linking against OpenSSL works... no

I'm determining my openssl location with:

[brad@reason Python-3.10.0]$ openssl version -d
OPENSSLDIR: "/etc/ssl"

To configure, I'm using:

./configure --with-openssl="/etc/ssl"

When I look for ssl.h, I find it in /usr/include/openssl. So I linked this directory to lib in /etc/ssl, but it was no help.

[brad@reason Python-3.10.0]$ ls -l /etc/ssl
total 40
lrwxrwxrwx 1 root root    16 Jul 16  2020 certs -> ../pki/tls/certs
-rw-r--r-- 1 root root   412 Oct 12 02:53 ct_log_list.cnf
-rw-r--r-- 1 root root   412 Oct 12 02:53 ct_log_list.cnf.dist
lrwxrwxrwx 1 root root    20 Oct 18 10:22 lib -> /usr/include/openssl
drwxr-xr-x 2 root root  4096 Oct 12 02:53 misc
-rw-r--r-- 1 root root 10909 Oct 12 02:53 openssl.cnf
-rw-r--r-- 1 root root 10909 Oct 12 02:53 openssl.cnf.dist
drwxr-xr-x 2 root root  4096 Oct 12 02:53 private
[brad@reason Python-3.10.0]$ sudo find / -name ssl.h | grep include
find: ‘/tmp/.mount_jetbraAJFEnl’: Permission denied
/home/brad/Downloads/freerdp-2.0.0-rc4/winpr/include/winpr/ssl.h
/home/brad/Downloads/FreeRDP/winpr/include/winpr/ssl.h
/home/brad/Development/tlscheck/openssl-1.1.1l/include/openssl/ssl.h
/usr/include/openssl/ssl.h
/var/lib/docker/overlay2/23e6f3c164ec8939352891c99393669df4ed6e66da1e04ce84616073f08c6051/diff/usr/include/openssl/ssl.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/18.08/c8075e929daaffcbe5c78c9e87c0f0463d75e90d2b59c92355fa486e79c7d0e3/files/include/nss/ssl.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/18.08/c8075e929daaffcbe5c78c9e87c0f0463d75e90d2b59c92355fa486e79c7d0e3/files/include/openssl/ssl.h
find: ‘/run/user/1000/gvfs’: Permission denied

This may be extraneous information, but my libssl.so is here:

[brad@reason Python-3.10.0]$ ls /usr/lib64 | grep ssl
libevent_openssl-2.0.so.5
libevent_openssl-2.0.so.5.1.9
libssl3.so
libssl.so
libssl.so.10
libssl.so.1.0.2k
openssl

Any thoughts on why make isn't able to include ssl, please let me know. Thanks.

user_10547390 · Answer

Had a very similar problem, with openssl not working and giving the same errors with python 3.10 on centos 7. Download openssl unpack then go to that directory

./config --prefix=/usr/local/custom-openssl --openssldir=/etc/ssl
make -j1 depend
make -j8
make install_sw

Then go to the python source unpack it and run in the directory

./configure -C --with-openssl=/usr/local/custom-openssl --with-openssl-rpath=auto --prefix=/usr/local/python-3.version
make -j8
make altinstall

See also Custom OpenSSL on https://docs.python.org/3/using/unix.html.

user_1959613 · Answer

This was very helpful, thanks! What kicked me onto this was the inability to run/update pip in a venv, so there wasn't much of an alternative to figuring out how to get OpenSSL to work.

As of this week (21 December 2021), in my environment (CentOS Linux release 7.9.2009 (Core)), I was able to shortcut parts of this.

From EPEL, installing openssl11, (yes, also openssl11-devel); to set up Python 3.10 properly I needed to create that "OpenSSL Directory" which I did by:

mkdir /usr/local/openssl11

Then setting symlinks for the (newly-installed) requirements:

cd /usr/local/openssl11
ln -s /usr/lib64/openssl11 lib
ln -s /usr/include/openssl11 include

And, finally, feeding this location to the configure script:

./configure --with-openssl=/usr/local/openssl11

Thanks for the really helpful explanations of how and why this was all necessary. Hope this helps others...

user_1056460 · Answer

As the output suggest, you need to point to openssl11. The easiest way I found to do it is to change the configure file in the python 3.10 source code directory:

$ sed -i 's/PKG_CONFIG openssl /PKG_CONFIG openssl11 /g' configure
$ sudo ./configure --enable-optimizations
.
.
checking whether compiling and linking against OpenSSL works... yes
.

Then you can do as @CristiFati suggested in order to check before installing that it is configured properly:

$ ./python -c "import sys, ssl; print(\"{:s}
{:s}\".format(sys.version, ssl.OPENSSL_VERSION))"
3.10.0 (default, Nov 29 2021, 17:48:34) [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
OpenSSL 1.1.1g FIPS  21 Apr 2020

user_3725508 · Answer

Python will use the bundled ssl (witch is outdated sometimes). In order to then use OpenSSL, add flag for OpenSSL when running ./configure. For detail, run ./configure --help for more options.

  --with-openssl=DIR      root of the OpenSSL directory
  --with-openssl-rpath=[DIR|auto|no]
                          Set runtime library directory (rpath) for OpenSSL
                          libraries, no (default): don't set rpath, auto:
                          auto-detect rpath from --with-openssl and
                          pkg-config, DIR: set an explicit rpath
  --with-ssl-default-suites=[python|openssl|STRING]
                          override default cipher suites string, python: use
                          Python's preferred selection (default), openssl:
                          leave OpenSSL's defaults untouched, STRING: use a
                          custom string, python and STRING also set TLS 1.2 as
                          minimum TLS version

user_4788546 · Answer

1. Intro

Some references that might be (more or less) useful:

[SO]: How to enable FIPS mode for libcrypto and libssl packaged with Python? (@CristiFati's answer)
[SO]: How to include OpenSSL in Visual Studio (@CristiFati's answer) (even if it's for Win, it has some good points that apply to Nix as well)
[SO]: OpenSSL FIPS_mode_set not working in Python cryptography library (@CristiFati's answer)

Working on a CentOS 7 image from Docker Hub (as it's closest to RHEL 7 (that you're having))

[root@cfati-5510-0:/work/q069539286]> uname -a
Linux cfati-5510-0 5.11.0-37-generic #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@cfati-5510-0:/work/q069539286]> cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@cfati-5510-0:/work/q069539286]> 
[root@cfati-5510-0:/work/q069539286]> rpm -qa | grep openssl
openssl-libs-1.0.2k-22.el7_9.x86_64
openssl-1.0.2k-22.el7_9.x86_64
openssl-devel-1.0.2k-22.el7_9.x86_64
[root@cfati-5510-0:/work/q069539286]> 
[root@cfati-5510-0:/work/q069539286]> openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
[root@cfati-5510-0:/work/q069539286]> which openssl
/usr/bin/openssl
[root@cfati-5510-0:/work/q069539286]> ldd /usr/bin/openssl
    linux-vdso.so.1 =>  (0x00007ffe101b7000)
    libssl.so.10 => /lib64/libssl.so.10 (0x00007f9483b8a000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f948393d000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f9483654000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f9483450000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f948321d000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f9482dba000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f9482bb6000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f94829a0000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f94825d2000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f94823c2000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f94821be000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f9481fa4000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9481d88000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9483dfc000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f9481b61000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f94818ff000)

2. OpenSSL considerations

[root@cfati-5510-0:/work/q069539286]> mkdir -p openssl-1.1.1g && cd openssl-1.1.1g
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> for g in /var/cache/yum/x86_64/7/epel/packages/openssl11-libs-1.1.1g-3.el7.x86_64.rpm /var/cache/yum/x86_64/7/epel/packages/openssl11-1.1.1g-3.el7.x86_64.rpm /var/cache/yum/x86_64/7/epel/packages/openssl11-devel-1.1.1g-3.el7.x86_64.rpm; do rpm2cpio ${g} | cpio -idm; done
7292 blocks
2123 blocks
8352 blocks
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls
etc  usr
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ll ./usr/bin/
total 637
-rwxrwxrwx 1 root root    610 Mar 29  2021 make-dummy-cert
-rwxrwxrwx 1 root root 644424 Mar 29  2021 openssl11
-rwxrwxrwx 1 root root    725 Mar 29  2021 renew-dummy-cert
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ./usr/bin/openssl11 
./usr/bin/openssl11: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ll ./usr/lib64/
total 3605
drwxrwxrwx 1 root root       0 Oct 21 23:55 engines-1.1
lrwxrwxrwx 1 root root      19 Oct 21 23:55 libcrypto.so.1.1 -> libcrypto.so.1.1.1g
-rwxrwxrwx 1 root root 3082216 Mar 29  2021 libcrypto.so.1.1.1g
lrwxrwxrwx 1 root root      16 Oct 21 23:55 libssl.so.1.1 -> libssl.so.1.1.1g
-rwxrwxrwx 1 root root  603568 Mar 29  2021 libssl.so.1.1.1g
drwxrwxrwx 1 root root       0 Oct 21 23:55 openssl11
drwxrwxrwx 1 root root       0 Oct 21 23:55 pkgconfig
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ldd ./usr/bin/openssl11 
    linux-vdso.so.1 =>  (0x00007ffca03f5000)
    libssl.so.1.1 => not found
    libcrypto.so.1.1 => not found
    libz.so.1 => /lib64/libz.so.1 (0x00007f3a61e56000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a61c52000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a61a36000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f3a61668000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3a6206c000)
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # Set LD_LIBRARY_PATH
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ldd ./usr/bin/openssl11
    linux-vdso.so.1 =>  (0x00007ffdf7fce000)
    libssl.so.1.1 => ./usr/lib64/libssl.so.1.1 (0x00007fe00f7a4000)
    libcrypto.so.1.1 => ./usr/lib64/libcrypto.so.1.1 (0x00007fe00f2c1000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fe00f0ab000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fe00eea7000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe00ec8b000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe00e8bd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe00fa34000)
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ./usr/bin/openssl11 version
OpenSSL 1.1.1g FIPS  21 Apr 2020
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ./usr/bin/openssl11 version -d
OPENSSLDIR: "/etc/pki/tls"

So, here's a working OpenSSL 1.1.1g FIPS version.

3. Python build

OpenSSL is fine from its PoV, but it needs some "small" adjustments in order for Python build to pick it up:

include and lib dirs must have the same parent (they reside lower in the dir tree, symlink them in the OpenSSL root dir)
Libraries with the 11 suffix (from earlier) must also be present, symlink them as well

[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # OpenSSL working. Prepare for Python build.
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # "include" and "lib" dirs must be at the same level - symlink them
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s ./usr/include/openssl11 ./include
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s ./usr/lib64/openssl11 ./lib
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls . ./lib
.:
etc  include  lib  usr

./lib:
libcrypto.so  libssl.so
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # "lib" dir must also contain $(LIBNAME).1.1 besides what's already in there ($(LIBNAME))
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s libcrypto.so ./lib/libcrypto.so.1.1
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s libssl.so ./lib/libssl.so.1.1
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls ./lib
libcrypto.so  libcrypto.so.1.1  libssl.so  libssl.so.1.1

Python (configure and make):

root@cfati-5510-0:/work/q069539286]> tar -zxf Python-3.10.0.tgz 
[root@cfati-5510-0:/work/q069539286]> cd Python-3.10.0
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./configure --with-openssl=$(pwd)/../openssl-1.1.1g --with-openssl-rpath=auto --prefix=$(pwd)/../python-3.10>../cfg.txt 2>&1
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> echo $?
0
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> LD_LIBRARY_PATH=$(pwd)/../openssl-1.1.1g/lib:${LD_LIBRARY_PATH} make -j8>../mk.txt 2>&1
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> echo $?
0

4. Test

Build was successful.
Quick tests (without make install):

[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # Quick test
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # --------------------------------
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./python -c "import sys, ssl; print(\"{:s}
{:s}\".format(sys.version, ssl.OPENSSL_VERSION))"
3.10.0 (default, Oct 22 2021, 01:12:00) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
OpenSSL 1.1.1g FIPS  21 Apr 2020
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # --------------------------------
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # As opposed to "regular" Python
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> python -c "import sys, ssl; print(\"{:s}
{:s}\".format(sys.version, ssl.OPENSSL_VERSION))"
2.7.5 (default, Nov 16 2020, 22:23:17) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
OpenSSL 1.0.2k-fips  26 Jan 2017

[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./python -c "import ssl; print(ssl._ssl)"

[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ldd /work/q069539286/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffef6b7b000)
    libssl.so.1.1 => /work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib/libssl.so.1.1 (0x00007f0ab9745000)
    libcrypto.so.1.1 => /work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib/libcrypto.so.1.1 (0x00007f0ab9262000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0ab9046000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f0ab8c78000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f0ab8a62000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f0ab885e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0ab9c05000)

How to compile python3 on RHEL with SSL? SSL cannot be imported

5 Answers

1. Intro

2. OpenSSL considerations

3. Python build

4. Test