Including non-Python files with setup.py

It is 2019, and here is what is working - despite advice here and there, what I found on the internet halfway documented is using setuptools_scm, passed as options to setuptools.setup. This will include any data files that are versioned on your VCS, be it git or any other, to the wheel package, and will make "pip install" from the git repository to bring those files along.

So, I just added these two lines to the setup call on "setup.py". No extra installs or import required:

    setup_requires=['setuptools_scm'],
    include_package_data=True,

No need to manually list package_data, or in a MANIFEST.in file - if it is versioned, it is included in the package. The docs on "setuptools_scm" put emphasis on creating a version number from the commit position, and disregard the really important part of adding the data files. (I can't care less if my intermediate wheel file is named "*0.2.2.dev45+g3495a1f" or will use the hardcoded version number "0.3.0dev0" I've typed in - but leaving crucial files for the program to work behind is somewhat important)

create MANIFEST.in in the project root with recursive-include to the required directory or include with the file name.

include LICENSE
include README.rst
recursive-include package/static *
recursive-include package/templates *

documentation can be found here

Step 1: create a MANIFEST.in file in the same folder with setup.py

Step 2: include the relative path to the files you want to add in MANIFEST.in

include README.rst
include docs/*.txt
include funniest/data.json

Step 3: set include_package_data=True in the setup() function to copy these files to site-package

Reference is here.

I wanted to post a comment to one of the questions but I don't enough reputation to do that >.>

Here's what worked for me (came up with it after referring the docs):

package_data={
    'mypkg': ['../*.txt']
},

include_package_data: False

The last line was, strangely enough, also crucial for me (you can also omit this keyword argument - it works the same).

What this does is it copies all text files in your top-level or root directory (one level up from the package mypkg you want to distribute).

Hope this helps!

This works in 2020!

As others said create "MANIFEST.in" where your setup.py is located.

Next in manifest include/exclude all the necessary things. Be careful here regarding the syntax. Ex: lets say we have template folder to be included in the source package.

in manifest file do this :

recursive-include template *

Make sure you leave space between dir-name and pattern for files/dirs like above. Dont do like this like we do in .gitignore

recursive-include template/* [this won't work]

Other option is to use include. There are bunch of options. Look up here at their docs for Manifest.in

And the final important step, include this param in your setup.py and you are good to go!

   setup(
    ...
    include_package_data=True,
    ......
)

Hope that helps! Happy Coding!

None of the above really worked for me. What saved me was this answer.
Apparently, in order for these data files to be extracted during installation, I had to do a couple of things:

1. Like already mentioned - Add a MANIFEST.in to the project and specify the folder/files you want to be included. In my case: recursive-include folder_with_extra_stuff *
2. Again, like already mentioned - Add include_package_data=True to your setup.py. This is crucial, because without it only the files that match *.py will be brought.
3. This is what was missing! - Add an empty __init__.py to your data folder. For me I had to add this file to my folder-with-extra-stuff.
4. Extra - Not sure if this is a requirement, but with my own python modules I saw that they're zipped inside the .egg file in site-packages. So I had to add zip_safe=False to my setup.py file.

Final Directory Structure

my-app/
├─ app/
│  ├─ __init__.py
│  ├─ __main__.py
├─ folder-with-extra-stuff/
│  ├─ __init__.py
│  ├─ data_file.json
├─ setup.py
├─ MANIFEST.in

None of the answers worked for me because my files were at the top level, outside the package. I used a custom build command instead.

import os
import setuptools
from setuptools.command.build_py import build_py
from shutil import copyfile

HERE = os.path.abspath(os.path.dirname(__file__))
NAME = "thepackage"

class BuildCommand(build_py):
    def run(self):
        build_py.run(self)

        if not self.dry_run:
            target_dir = os.path.join(self.build_lib, NAME)
            for fn in ["VERSION", "LICENSE.txt"]:
                copyfile(os.path.join(HERE, fn), os.path.join(target_dir,fn))

 
 
setuptools.setup(
    name=NAME,
    cmdclass={"build_py": BuildCommand},
    description=DESCRIPTION,
    ...
)

For non-python files to be included in an installation, they must be within one of the installed package directories. If you specify non-python files outside of your package directories in MANIFEST.in, they will be included in your distribution, but they will not be installed. The "documented" ways of installing arbitrary files outside of your package directories do not work reliably (as everyone has noticed by now).

The above answer from Julian Mann copies the files to your package directory in the build directory, so it does work, but not if you are installing in editable/develop mode (pip install -e or python setup.py develop). Based on this answer to a related question (and Julian's answer), below is an example that copies files to your installed package location either way after all the other install/develop tasks are done. The assumption here is that files file1 and file2 in your root-level data directory will be copied to your installed package directory (my_package), and that they will be accessible from python modules in your package using os.path.join(os.path.dirname(__file__), 'file1'), etc.

Remember to also do the MANIFEST.in stuff described above so that these files are also included in your distribution. Why setuptools would include files in your distribution and then silently never install them, is beyond my ken. Though installing them outside of your package directory is probably even more dubious.

import os
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
from shutil import copyfile

HERE = os.path.abspath(os.path.dirname(__file__))
NAME = 'my_package'

def copy_files (target_path):
    source_path = os.path.join(HERE, 'data')
    for fn in ["file1", "file2"]:
        copyfile(os.path.join(source_path, fn), os.path.join(target_path,fn))

class PostDevelopCommand(develop):
    """Post-installation for development mode."""
    def run(self):
        develop.run(self)
        copy_files (os.path.abspath(NAME))

class PostInstallCommand(install):
    """Post-installation for installation mode."""
    def run(self):
        install.run(self)
        copy_files (os.path.abspath(os.path.join(self.install_lib, NAME)))

setup(
    name=NAME,
    cmdclass={
        'develop': PostDevelopCommand,
        'install': PostInstallCommand,
    },
    version='0.1.0',
    packages=[NAME],
    include_package_data=True,
    setup_requires=['setuptools_scm'],
)