PyInstaller, how to include data files from an external package that was installed by pip?

Viewed 10218

Problem

I am attempting to use PyInstaller to create an application for internal use within my company. The script works great from a working python environment, but loses something when translated to a package.

I know how to include and reference data files that I myself need within my package, but I am having trouble including or referencing files that should come in when imported.

I am using a pip-installable package called tk-tools, which includes some nice images for panel-like displays (looks like LEDs). The problem is that when I create a pyinstaller script, any time that one of those images is referenced, I get an error:

DEBUG:aspen_comm.display:COM23 19200
INFO:aspen_comm.display:adding pump 1 to the pump list: [1]
DEBUG:aspen_comm.display:updating interrogation list: [1]
Exception in Tkinter callback
Traceback (most recent call last):
  File "tkinter\__init__.py", line 1550, in __call__
  File "aspen_comm\display.py", line 206, in add
  File "aspen_comm\display.py", line 121, in add
  File "aspen_comm\display.py", line 271, in __init__
  File "aspen_comm\display.py", line 311, in __init__
  File "lib\site-packages\tk_tools\visual.py", line 277, in __init__
  File "lib\site-packages\tk_tools\visual.py", line 289, in to_grey
  File "lib\site-packages\tk_tools\visual.py", line 284, in _load_new
  File "tkinter\__init__.py", line 3394, in __init__
  File "tkinter\__init__.py", line 3350, in __init__
_tkinter.TclError: couldn't open "C:\_code\tools\python\aspen_comm\dist\aspen_comm\tk_tools\img/led-grey.png": no such file or directory

I looked within that directory in the last line - which is where my distribution is located - and found that there is no tk_tools directory present.

Question

How to I get pyinstaller to collect the data files of imported packages?

Spec File

Currently, my datas is blank. Spec file, created with pyinstaller -n aspen_comm aspen_comm/__main__.py:

# -*- mode: python -*-

block_cipher = None


a = Analysis(['aspen_comm\\__main__.py'],
             pathex=['C:\\_code\\tools\\python\\aspen_comm'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)

pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='aspen_comm',
          debug=False,
          strip=False,
          upx=True,
          console=True )

coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='aspen_comm')

When I look within /build/aspen_comm/out00-Analysis.toc and /build/aspen_comm/out00-PYZ.toc, I find an entry that looks like it found the tk_tools package. Additionally, there are features of the tk_tools package that work perfectly before getting to the point of finding data files, so I know that it is getting imported somewhere, I just don't know where. When I do searches for tk_tools, I can find no reference to it within the file structure.

I have also tried the --hidden-imports option with the same results.

Partial Solution

If I 'manually' add the path to the spec file using datas = [('C:\\_virtualenv\\aspen\\Lib\\site-packages\\tk_tools\\img\\', 'tk_tools\\img\\')] and datas=datas in the Analysis, then all works as expected. This will work, but I would rather PyInstaller find the package data since it is clearly installed. I will keep looking for a solution, but - for the moment - I will probably use this non-ideal workaround.

If you have control of the package...

Then you can use stringify on the subpackage, but this only works if it is your own package.

7 Answers

Here's a one-liner using the same idea as Turn mentioned. In my case I needed a package (zbarcam) that was inside of kivy_garden. But I tried to generalize the process here.

from os.path import join, dirname, abspath, split
from os import sep
import glob
import <package>

pkg_dir = split(<package>.__file__)[0]
pkg_data = []
pkg_data.extend((file, dirname(file).split("site-packages")[1]) for file in glob.iglob(join(pkg_dir,"**{}*".format(sep)), recursive=True))

Possibly this option was added after this question was asked, but PyInstaller provides a few command line options that helped:

--collect-submodules MODULENAME

Collect all submodules from the specified package or module. This option can be used multiple times.

--collect-data MODULENAME, --collect-datas MODULENAME

Collect all data from the specified package or module. This option can be used multiple times.

--collect-binaries MODULENAME

Collect all binaries from the specified package or module. This option can be used multiple times.

--collect-all MODULENAME

Collect all submodules, data files, and binaries from the specified package or module. This option can be used multiple times.

I simply used the name of the external package as MODULENAME

The following code will put all the PNG files in your directory into a folder in the top level of the bundled app called imgs:

datas=[("C:\\_code\\tools\\python\\aspen_comm\\dist\\aspen_comm\\tk_tools\\img\\*.png", "imgs")],

You can then reference them with os.path.join("imgs", "your_image.png") in your code.

Edited to Add

To solve this problem more permanently, I created a pip-installable package called stringify which will take a file or directory and convert it into a python string so that packages such as pyinstaller will recognize them as native python files.

Check out the project page, feedback is welcome!


Original Answer

The answer is a bit roundabout and deals with the way that tk_tools is packaged rather than pyinstaller.

Someone recently made me aware of a technique in which binary data - such as image data - could be stored as a base64 string:

with open(img_path, 'rb') as f:
    encoded_string = base64.encode(f.read())

The encoded string actually stores the data. If the originating package simply stores the package files as strings instead of as image files and creates a python file with that data accessible as a string variable, then it is possible to simply include the binary data within the package in a form that pyinstaller will find and detect without intervention.

Consider the below functions:

def create_image_string(img_path):
    """
    creates the base64 encoded string from the image path 
    and returns the (filename, data) as a tuple
    """

    with open(img_path, 'rb') as f:
        encoded_string = base64.b64encode(f.read())

    file_name = os.path.basename(img_path).split('.')[0]
    file_name = file_name.replace('-', '_')

    return file_name, encoded_string


def archive_image_files():
    """
    Reads all files in 'images' directory and saves them as
    encoded strings accessible as python variables.  The image
    at images/my_image.png can now be found in tk_tools/images.py
    with a variable name of my_image
    """

    destination_path = "tk_tools"
    py_file = ''

    for root, dirs, files in os.walk("images"):
        for name in files:
            img_path = os.path.join(root, name)
            file_name, file_string = create_image_string(img_path)

            py_file += '{} = {}\n'.format(file_name, file_string)

    py_file += '\n'

    with open(os.path.join(destination_path, 'images.py'), 'w') as f:
        f.write(py_file)

If archive_image_files() is placed within the setup file, then the <package_name>/images.py is automatically created anytime the setup script is run (during wheel creation and installation).

I may improve on this technique in the near future. Thank you all for your assistance,

j

Mac OS X 10.7.5 Pyinstaller; adding images using 1 line of code, instead of individual lines, for each image in the app.spec file. This is all the code I used to get my images to compile with my script. Add this function to the top of: yourappname.py:

# Path to the resources, (pictures and files) needed within this program def resource_path(relative_path): """ Get absolute path to resource, works for dev and for PyInstaller """ try:

# PyInstaller creates a temp folder and stores path in _MEIPASS base_path = sys._MEIPASS

   except Exception:
        base_path = os.path.abspath(".")
        return os.path.join(base_path, relative_path)`  

Also, in the appname.py script, add this 'resource_path' to get your images from your resource, like this:

yourimage = PhotoImage(file=resource_path("yourimage.png"))

In your appname.spec file replace 'datas=[], with your pathway to the images, you want to use. I used only '*.png' image files and this worked for me:

datas=[("/Users/rodman/PycharmProjects/tkinter/*.png", ".")],

Make sure you replace /Users/rodman/PycharmProjects/tkinter/ with your path, to the folder where your images are located. Forgive the sloppy code formmating, I am not used to these code tags, and thanks Steampunkery, for stearring me in the right direction to figure this Mac os x answer.

A bit late to the party, but wrote a help article on how I did this:

https://www.linkedin.com/pulse/loading-external-module-data-during-pyinstaller-bundling-deguzis/?published=t

Snippet:

import os
import pkgutil
import PyInstaller.__main__
import platform
import shutil
import sys


# Get mypkg data not imported automagically
# Pre-create location where data is expected
if not os.path.exists('ext_module'):
    os.mkdir('ext_module')

with open ('ext_module' + os.sep + 'some-env.ini', 'w+') as f:
    data = pkgutil.get_data( 'ext_module', 'some-env.ini' ).decode('utf-8', 'ascii')
    f.write(data)

# Set terminator (PyInstaller does not provide an easy method for this)
# ':' for OS X / Linux
# ';' for Windows
if platform.system() == 'Windows':
    term = ';'
else:
    term = ':'


PyInstaller.__main__.run([
        '--name=%s' % 'mypkg',
        '--onefile',
        '--add-data=%s%smypkg' % (os.path.join('mypkg' + os.sep + 'some-env.ini'),term),
        os.path.join('cli.py'),
    ])


# Cleanup
shutil.rmtree('mypkg')
Related