I have a Python project folder which looks something like this:
|- main.py
|- data
|- out
|- tests
|- src
|- utils
|- etc...
Now imagine that inside the source code in src I need to continuously reference to the data folder (e.g. I have some dataset in a fixed location to load with a data loader in utils), or the output folder. What is the best way to go about this?
i) Not working solution
The first solution that comes to mind is to do something like:
data_dir = './data'
with open(data_dir + 'file.txt') ...
The reference is relative to main.py, since I always expect code to be launched from there. However, this is not always the case, as I could for example launch a test from the relative folder.
ii) Working inelegant solution
The second solution is to, in every file in src, do something like
import os
this_file = os.path.abspath(__file__)
and from here reference the folder you need relative to the file, not depending on where it was launched from (test, main, etc...). This is ok, but I feel like it's not very elegant, requires too many lines of code and need to be changed in every file if the data folder changes position (though, I assume pretty static folder organisation for now).
iii) My solution
My current solution is the following: create a file src/utils/project_vars.py which only contains the absolute paths to some fixed folders. It looks something like:
import os
# use to find all absolute paths relative to this file
_file_abspath = os.path.abspath(__file__)
# absolute paths of some folders
SRC_DIR = os.path.dirname(os.path.dirname(_file_abspath))
DATA_DIR = os.path.join(
os.path.dirname(SRC_DIR),
'data'
)
OUT_DIR = os.path.join(
os.path.dirname(SRC_DIR),
'out'
)
To find the data folder inside any script in the project I just need to import the DATA_DIR variable from here and then use it. Another important advantage is that this solution can be used across platform, e.g. on the cluster I run computations on as well as in local.
Question: is there a way to handle this kind of situations organically, such that references remain solid across platforms? Are there common patterns that are usually employed?