Common Python scripts are getting called multiple times

user_15710932 0
Viewed 383

I have the following project structure.

 - README.rst
 - LICENSE
 - setup.py
 - requirements.txt
 - common_prj/__init__.py
 - common_prj/common_lib.py
 - common_prj/config.xml
 - Project1/__init__.py
 - Project1/some_app_m1.py
 - Project1/some_app_m2.py
 - Project1/some_app.py
 - Project2/some_app1.py

I am having all my common classes in common_prj/common_lib.py file. Now each module file from Project1 is calling common_prj/common_lib.py to access common classes and setup project env using config.xml

#import in some_app_m1.py

from common_prj import common_lib

#import in some_app_m2.py

from common_prj import common_lib

Imports in some_app.py

from Project1 import some_app_m1
from Project1 import some_app_m2 
from common_prj import common_lib
###
  <some functions>
  <some functions>
###
if __name__ == '__main__':

With the above three import it seems common_lib.py is getting executed multiple time , but if I keep the common_lib.py in Project1 then I don't see this issue.

Please let me now how can I keep common_lib.py in common package and call that from Project1 scripts without executing it multiple time. The purpose of common_lib.py is to share common classes with scripts in Project2.

I have the below code in common_lib.py which is repeating for calling classes from each module after import.

self.env = dict_env.get(int(input("Choose Database ENV for this execution : \n" + str(dict_env) + "\nSelect the numeric value => ")))
self.app = dict_app.get(int(input("Choose application for this execution : \n" + str(dict_app) + "\nSelect the numeric value => ")))

My question why I am not facing this issue if I keep common_lib.py in Project1 rather common_prj. I added the above code in common_lib.py because I don't wanted to repeat these lines and ENV setup in my all app code. These are global env settings for all application scripts code inside Project1.

output

Choose Database ENV for this execution : 
{1: 'DEV', 2: 'SIT', 3: 'UAT', 4: 'PROD'}
Select the numeric value => 1
Choose application for this execution : 
{1: 'app1', 2: 'app2', 3: 'app3', 4: 'app4', 5: 'app5', 6: 'app6'}
Select the numeric value => 1
Choose Database ENV for this execution : ### Here it is repeating again from common_lib.py
{1: 'DEV', 2: 'SIT', 3: 'UAT', 4: 'PROD'}
Select the numeric value => 1
Choose application for this execution : ### Here it is repeating again from common_lib.py
{1: 'app1', 2: 'app2', 3: 'app3', 4: 'app4', 5: 'app5', 6: 'app6'}
Select the numeric value => 1
python
3 Answers

Note: This is my first answer where I answer in-depth of the logistics of python. If I have said anything incorrect, please let me know or edit my answer (with a comment letting me know why). Please feel free to clarify any questions you may have too.

As the comments have discussed if __name__ == '__main__': is exactly the way to go. You can check out the answers on the SO post for a full description or check out the python docs.

In essence, __name__ is a variable for a python file. Its behavior includes:

  • When used in a file, its value is defaulted to '__main__'.
  • When a file imports another program. (import foo). foo.__name__ will return 'foo' as its value.

Now, when importing a file, python compiles the new file completely. This means that it will run everything that can be run. Python compiles all the methods, variables, etc. As it's compiling, if python comes across a callable function (i.e hello_world()) it will run it. This happens even if you're trying to import a specific part of the package or not.

Now, these two things are very important. If you have some trouble understanding the logic, you can take a look at some sample code I created.

Since you are importing from common_prj 3 times (from the some_app_m* file and my_app.py file), you are running the program in common_prj three times (This is also what makes python one of the slower programming languages).

Although I'm unable to see your complete code, I'm assuming that common_prj.py calls common_lib() at some point in your code (at no indentation level). This means that the two methods inside:

self.env = dict_env.get(int(input("Choose Database ENV for this execution : \n" + str(dict_env) + "\nSelect the numeric value => ")))
self.app = dict_app.get(int(input("Choose application for this execution : \n" + str(dict_app) + "\nSelect the numeric value => ")))

were also called.

So, in conclusion:

  1. You imported common_prj 3 times
  2. Which ran common_lib multiple times as well.

Solution:

  • This is where the __name__ part I talked about earlier comes into play
  • Take all callable functions that shouldn't be called during importing, and place them under the conditional of if __name__ == '__name__':
  • This will ensure that only if the common_prj file is run then the condition will pass.

Again, if you didn't understand what I said, please feel free to ask or look at my sample code!

Edit: You can also just remove these callable functions. Files that are meant to be imported use them to make sure that the functions work as intended. If you look at any of the packages that you import daily, you'll either find that the methods are called in the condition OR not in the file at all.

user_111777200

This is more of a question of library design, rather than the detailed mechanics of Python.

A library module (which is what common_prj is) should be designed to do nothing on import, only define functions, classes, constants etc. It can do things like pre-calculating constants, but it should be restricted to things that don't depend on anything external, produce the same results each time and don't involve much heavy computation. At that point, the code running three times will no longer be a problem.

If the library needs parameters, the caller should supply those, ideally in a way that allows multiple different sets of parameters to be used within the same program. Depending on the situation, the parameters can be a main object that manages the rest of the interaction, or parameters that are passed into each object constructor or function call, or a configuration object that's passed in. The library can provide a function to gather these parameters from the environment or user input, but it shouldn't be required; it should always be possible for the program to supply the parameters directly. This will also make tests easier to write.

Side notes:

  • The name "common" is a bit generic; consider renaming the library after what it does, or splitting it into several libraries each of which does one thing?
  • Python will try to cache imported modules so it doesn't re-run them multiple times; this is probably why it only ran the code once in one situation but three times in another. This is another reason to have only definitions happen on import, so the details of the caching don't affect functionality.
user_6833290

It seems after commenting the below in

Project1/init.py

the issue has been resolved. I was calling the same common_lib from init.py as well which was causing this issue . Removing the below entry resolved the issue.

#from common_prj import commonlib
user_157109320
Related