Organize python scripts for a data science project

user_454671 0
Viewed 200

I have a folder with many subfolders and each subfolder has (n) python scripts that do a variety of tasks for the project. It could be data analysis, call other scripts, automate some stuff etc. Some of these scripts are related to each other, some are standalone. But they are all part of the same 'project'.

I come from Java world and I am used to packaging everything related to a project in a .jar file. Is there something similar I can do to organize these wild python scripts, even to the point of giving a common entry point into all of them?

devops python
2 Answers

Yes, you can package the project and install it. I like using Poetry for dependency management, virtualenvs, and packaging. It can install your project either in a virtual or global environment, or build a *.whl for your project that makes it pip installable elsewhere (in a docker container, cloud resource, etc.). It's sort of like a Maven for Python.

As for the "wild scripts", there's no reason why your python code has to be disorganized. All of your usual hygiene around clean code, modular OO design patterns, good encapsulation, dependency injection, etc. are still encouraged; python just won't force those guard rails onto you like Java so it's very much "bring your own good habits". I often organize my python project into modular java-esque subpackages where my domain models and other reusable components are defined. These can then be imported in any scripts I write in my scripts folder. This also makes the scripts themselves quite a bit more maintainable and orderly.

A rough example structure could be (depending on the type of project)

project-root
   - domain
     - domain_model_a
     - domain_model_b
   - training
     - machine_learning_model
   - storage
     - repository
        - domain_model_a_repository
        - domain_model_b_repository
     - service
        - elastic_search_service
   - script
        - script_that_does_X
        - ml_training_script

Lastly, it can be nice for your dev workflow to toss in if __name__ == "__main__": scripts at the bottom of your components if you want to be able to jump into and invoke, say, a repository or service as a script for debugging purposes and inform what kinds of integration tests you want to write.

user_53233110

You may want to have a look at this cookiecutter for data science projects.

user_60473570
Related