How are Python's C and C++ libraries cross platform?

user_7092900 0
Viewed 817

Many of Python's libraries, e.g. Pandas and Numpy, are actually C or C++ with Python wrappers round them. I have no experience with compiled languages and don't understand how these libraries are cross platform (i.e. run on Mac, Windows, Linux), since my understanding is that C and C++ need to be compiled for a specific operating system. How does this work?

Edits:
How do you compile Python C/C++ extensions for different OS/versions of Python? does not answer my question and therefore this is not a duplicate. This question is about understanding how it works, that question presumes this understanding and is about implementation.

c python c++
1 Answers

As has been pointed out in comments, Python package using C/C++ compiled code require compilation on the target architecture for them to be cross-platform.

Under the hood, when you use pip install pandas for example, pip will look for the requested package on PyPI and, if available, it will install the wheel corresponding to your specific system. A wheel is a distribution mechanism that helps installation of python packages on specific python distribution and/or target architecture. Taking the example of pandas again, here is what an upgrade of pandas returned this morning:

applepie:~ applepie$ pip install pandas --upgrade
Collecting pandas
  Downloading pandas-1.1.3-cp38-cp38-macosx_10_9_x86_64.whl (10.1 MB)
     |████████████████████████████████| 10.1 MB 7.2 MB/s 
Requirement already satisfied, skipping upgrade: numpy>=1.15.4 in ./.pyenv/versions/3.8.5/lib/python3.8/site-packages (from pandas) (1.19.2)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in ./.pyenv/versions/3.8.5/lib/python3.8/site-packages (from pandas) (2020.1)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in ./.pyenv/versions/3.8.5/lib/python3.8/site-packages (from pandas) (2.8.1)
Requirement already satisfied, skipping upgrade: six>=1.5 in ./.pyenv/versions/3.8.5/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)
Installing collected packages: pandas
  Attempting uninstall: pandas
    Found existing installation: pandas 1.1.2
    Uninstalling pandas-1.1.2:
      Successfully uninstalled pandas-1.1.2
Successfully installed pandas-1.1.3

Note that the first step executed was to download a .whl file that matches my specific architecture (Mac OSX, x86_64). The file name has further information, for example it is pandas v 1.1.3 and is compatible with CPython 3.8. Running this command on a different machine would have yielded a different output.

You can view the list of available files for pip to look for directly on PyPI. Again, looking for pandas on PyPI shows that the most up-to-date wheel for Mac OSX on CPython 3.8 is named pandas-1.1.3-cp38-cp38-macosx_10_9_x86_64.whl, which unsurprisingly is what pip install pandas --upgrade downloaded and installed.

I am no expert in python distribution, in fact I was only very recently introduced to python wheels, have never distributed python code and had to do some reading prior to answering this question, however it is my understanding that Python packages with C/C++ components would first require compilation on each architecture and then to build a specific wheel for that combination of python version and computer architecture. If a compatible wheel is not found, installing a Python package with C/C++ may require compilation.

user_14572690
Related