Want to know how packages are managed in Conda Package Manager

user_14101583 0
Viewed 49

I am new to Data Sciences. Online resources suggest that most data scientists use conda + pip to manage their packages and dependencies, so I also did the same. Let me take a simple example where I just have a single virtual environment named virtualEnv. So essentially conda has 2 environments now :

  • base
  • virtualEnv

Now let's say that I have installed PyTorch=1.9.0 in my base environment. Now if I switch to virtualEnv and try to install PyTorch=1.9.0 (the same version as that of base) in that environment, then what happens?

Does conda pickup the PyTorch packages from the base environment or does it install the package from its online repo again.

Also, I installed PyTorch in base env using this command : provided on PyTorch website

 conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch

So do I install PyTorch in virtualEnv using the same command or just the simple command would work:

conda install pytorch cudatoolkit torchvision torchaudio

Basically the -c pytorch is the difference between the two commands.

I tried to find this on google but couldn't find anything.

data-science anaconda conda package-managers python
2 Answers

I think it will pull the dependencies from the online repository, using local dependencies happen when you are cloning the environemnt - see --clone parameter (used during conda create).

The -c gives an information about the channel from which these packages should come from - -c torch most probably will download them directly from the pytorch repositories - without this option the package manager might try to use a different source and not find for example the version that you were looking for.

In the second call you didn't specify the version of the cuda toolkit so there is a risk that it will not be compatible with the downloaded torch - I would suggest to download it as it's specified on the page.

Didn't use conda for a while - this is what I remembered.

user_144608660

First of all, conda itself is installed in the base environment. For the purpose of tidyness it is a good practice to avoid using it for anything else but rather create new environments and install the packages you need in those. (As stated in the conda docs)

So in your example you would ideally install pytorch in your virtualEnv.

You don't have to worry about flooding your disk by installing packages twice. Conda uses hard links if it detects that it already has the same version of the package installed instead of saving the same thing twice.

The -c part of your installation command selects the channel from which to install the package. You can think of channels like links to repositories.

The pytorch channel is managed by the pytorch organisation and thus probably has the most recent version of the pytorch packages.

You can see which packages are available on which channel on the anaconda page

user_165601080
Related