Python: Where to put external packages?

user_5115219 0
Viewed 648

I have a Django project that uses an external package that I haven't installed via pip in my virtualenv, rather I got it via git clone, and now I am wondering where to put it, and how the name of the folder should be. The external package is this: Evernote Python3 SDK

This is my current Django project structure:

➜  (venv:evernote) evernote_bear_project git:(master) tree
.
├── README.md
├── bear2evernote
│   ├── static
│   └── templates
├── config
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── libs
│   ├── evernote-sdk-python3

│       (omitted)

│   └── utils
│       ├── __init__.py
│       └── file_utils.py
└── manage.py

I have put this package evernote-sdk-python3 in a folder called libs in my Django project folder. I also put utility functions in a package called utils beneath libs

Now I know that there are many suggestions out there on how to structure your projects like these two:

[1] Structuring Your Project — The Hitchhiker's Guide to Python

[2] Python Application Layouts: A Reference – Real Python

And I also know that Python does not force you to use a dogmatic folder structure. But nontheless: I don't have a solid understanding of packaging, package naming and package structuring.

  1. Is it good practice to put an external package underneath my (Django)project root folder?

  2. Does it make sense to put all my packages / libraries, that are not related to my specific Django apps, in one single folder named libs or utils or src?

Edit: I also do not see a clear best practice on how to use these folder names:

  • src
  • lib
  • utils
django python
1 Answers

In general, the recommended workflow is the following:

  1. Your virtualenv should already be excluded from version control (e.g. with .gitignore or .hgignore). If it's not, do so now.
  2. Where possible, install things into your virtualenv from PyPI rather than by downloading them from GitHub or elsewhere.
  3. If you must install something not on PyPI, pass a URL like git://... to pip install, using one of the forms shown in the reference manual. If you need to pin to a specific commit, pass -e to install as well, and include a commit identifier in the URL. Otherwise, the dependency will effectively live "at head" and you may not be able to get back to that exact commit in the future.
  4. If for some reason you must maintain a local fork of someone else's project (e.g. because you are running 2to3 over it?), then it may make sense to do as you propose. Install with pip install -e ./libs/libfoo after cloning (or downloading, as the case may be). This is not recommended as you will need to periodically pull in or backport changes from upstream, which is very labor-intensive. Version control also becomes a pain with this approach because you may not have the upstream history available for 3-way merging (there are ways around this - note the commands Linus shows may not be the best way of doing it today).
  5. Once you have everything you need installed, use pip freeze to generate a requirements.txt file, stick said text file into your Django project directory, and then check it into version control. This file should be sufficient to completely regenerate all of your dependencies from their upstream sources without needing to check any of your dependencies into source control (unless you are doing #4, of course).
  6. Periodically do pip install -U and run regression tests against the newer versions of your dependencies, then redo the pip freeze and check the changes into version control. You can also manually edit the requirements.txt file as needed when dependencies break backwards compatibility or otherwise cause problems.
user_13403890
Related