I have a machine learning job in prod that keeps failing because of the reason below:
InvalidConfigurationError: You haven't configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/PythonShell.py configure`
To address it, I then installed the packages on the workers by doing:
%pip install
as shown in https://community.databricks.com/s/feed/0D53f00001HKIENCA5, instead of
%sh
pip install
The new problem that I now have is that '%pip' restarts the python interpreter and I do not want that because I lose any python computing done before the command.
Question: How do I install a package on the workers without restarting the python interpreter?