Python: why does importing a package sometimes grant access to modules underneath it but sometimes not?

Question

Python import mechanism is always a myth for me. Sometimes importing a package can grant access to the modules underneath it. For instance,

import urllib
urllib.parse.unquote

gives

which shows functions are accessible even with only the package (i.e. urllib in this case) imported but not down to the module file. This is done within Jupyter notebook.

But when I do the same thing in terminal

>>> import urllib
>>> urllib.parse.unquote
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: module 'urllib' has no attribute 'parse'

Both Python versions are 3.6.1.

What makes the difference and what is the good practice?

EDIT to combine the answers by @user2357112 and @Tomoki.

Directly from @user2357112

For an access to urllib.parse to work, the following two conditions must be true:

The urllib module object must be bound to the urllib name, whether in the local, global, or some enclosing scope. The urllib.parse submodule must have been initialized and bound to the parse attribute of the urllib module object. An import urllib in the current local or global scope (or any enclosing scope) satisfies the first condition.

An import urllib.parse executed anywhere in the program satisfies the second condition, since it loads the submodule and binds it to the parse attribute on the urllib module object, and there's only one urllib module object for the whole program.

In the environments where urllib.parse was accessible after a simple import urllib, some other code must have loaded urllib.parse, causing you to see it.

The evidence is provided by @Tomoki

Test: "import IPython"
 └─IPython:┐
      ┌────┘
      ├──"from core.application import Application"
      │   └──IPython.core.application: "from IPython.core import release, crashhandler"
      │      └──IPython.core.crashhandler: "from IPython.core import ultratb"
      │         └──IPython.core.ultratb: "import pydoc"
      │            └──pydoc: "import urllib.parse"
      └──"from terminal.embed import embed"
          └──IPython.terminal.embed:┐
                        ┌───────────┘
                        ├──"from IPython.core import magic_arguments"
                        │   └──IPython.core.magic_arguments: "from IPython.utils.text import dedent"
                        │      └──IPython.utils.text: "from pathlib import Path"
                        │         └──pathlib: "from urllib.parse import quote_from_bytes"
                        ├──"from IPython.core.magic import Magics, magics_class, line_magic"
                        │   └──IPython.core.magic: "from IPython.core import oinspect"
                        │      └──IPython.core.oinspect: "from IPython.core import page"
                        │         └──IPython.core.page: "from IPython.core.display import display"
                        │            └──IPython.core.display: "import mimetypes"
                        │               └──mimetypes: "import urllib.parse"
                        └──"from IPython.terminal.interactiveshell import TerminalInteractiveShell"
                            └──pygments.plugin: "import pkg_resources"
                               └──pkg_resources: "import email.parser"
                                  └──email.parser: "from email.feedparser import FeedParser, BytesFeedParser"
                                     └──email.feedparser: "from email._policybase import compat32"
                                        └──email._policybase: "from email.utils import _has_surrogates"
                                           └──email.utils: "import urllib.parse"

The last line indeed touches urllib.parse.

Another Evidence

import scipy does not provide access to scipy.stats.norm in either a terminal or Jupyter notebook because none of the environment touches scipy.stats at all.

What is the good practice?

We can conclude from above that it is not only a good practice, but in fact a requirement to ##import the whole module levels##.

"Always import down to file (module) level to guarantee the access"

Thanks all for the answers!

user_2357112 · Answer

For an access to urllib.parse to work, the following two conditions must be true:

The urllib module object must be bound to the urllib name, whether in the local, global, or some enclosing scope.
The urllib.parse submodule must have been initialized and bound to the parse attribute of the urllib module object.

An import urllib in the current local or global scope (or any enclosing scope) satisfies the first condition.

An import urllib.parse executed anywhere in the program satisfies the second condition, since it loads the submodule and binds it to the parse attribute on the urllib module object, and there's only one urllib module object for the whole program.

In the environments where urllib.parse was accessible after a simple import urllib, some other code must have loaded urllib.parse, causing you to see it.

user_8223843 · Answer

Like user2357112 said it's getting imported; I believe these are the specific modules and statements.

Test: "import IPython"
 └─IPython:┐
      ┌────┘
      ├──"from core.application import Application"
      │   └──IPython.core.application: "from IPython.core import release, crashhandler"
      │      └──IPython.core.crashhandler: "from IPython.core import ultratb"
      │         └──IPython.core.ultratb: "import pydoc"
      │            └──pydoc: "import urllib.parse"
      └──"from terminal.embed import embed"
          └──IPython.terminal.embed:┐
                        ┌───────────┘
                        ├──"from IPython.core import magic_arguments"
                        │   └──IPython.core.magic_arguments: "from IPython.utils.text import dedent"
                        │      └──IPython.utils.text: "from pathlib import Path"
                        │         └──pathlib: "from urllib.parse import quote_from_bytes"
                        ├──"from IPython.core.magic import Magics, magics_class, line_magic"
                        │   └──IPython.core.magic: "from IPython.core import oinspect"
                        │      └──IPython.core.oinspect: "from IPython.core import page"
                        │         └──IPython.core.page: "from IPython.core.display import display"
                        │            └──IPython.core.display: "import mimetypes"
                        │               └──mimetypes: "import urllib.parse"
                        └──"from IPython.terminal.interactiveshell import TerminalInteractiveShell"
                            └──pygments.plugin: "import pkg_resources"
                               └──pkg_resources: "import email.parser"
                                  └──email.parser: "from email.feedparser import FeedParser, BytesFeedParser"
                                     └──email.feedparser: "from email._policybase import compat32"
                                        └──email._policybase: "from email.utils import _has_surrogates"
                                           └──email.utils: "import urllib.parse"

user_7660234 · Answer

Python 3 does not load the helper modules for urllib automatically. ( https://docs.python.org/2/library/urllib.html )

"Note The urllib module has been split into parts and renamed in Python 3 to urllib.request, urllib.parse, and urllib.error. The 2to3 tool will automatically adapt imports when converting your sources to Python 3."

"Note urllib also exposes certain utility functions like splittype, splithost and others parsing URL into various components. But it is recommended to use urlparse for parsing URLs rather than using these functions directly. Python 3 does not expose these helper functions from urllib.parse module."

If you attempt to query the urllib namespace dir(urllib) after import, there are no submodules. After you type urllib.parse.unquote and get the error, the urllib helper modules are loaded. (I'm serious, that sounds crazy and wrong, all things not Python, "he's a n00b", just try it.) You can see them in the namespace through dir(urllib) and can query use them as if they were all loaded initially. You will then get the function object return.

Python 3.5.2 (default, Aug 18 2017, 17:48:00) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information.

>>> import urllib

>>> urllib.parse.unquote

Traceback (most recent call last): File "", line 1, in AttributeError: module 'urllib' has no attribute 'parse'

>>> urllib.parse.unquote

In the six module, there are builtins.module(builtins.object)

    Module_six_moves_urllib
_LazyDescr(builtins.object)
    MovedAttribute
    MovedModule
_LazyModule(builtins.module)
    Module_six_moves_urllib_error
    Module_six_moves_urllib_parse
    Module_six_moves_urllib_request
    Module_six_moves_urllib_response
    Module_six_moves_urllib_robotparser

There is additional documentation (of course) such as

class Module_six_moves_urllib(builtins.module) " | Create a six.moves.urllib namespace that resembles the Python 3 namespace"

I suspect that terminal does not invoke the builtin to load the helper modules automatically where as Jupyter does though I honestly don't know.

Edit to add: Importing urllib, importing six and calling on it [even help("six")] will load the parse, request, response, robotparser modules to the urllib namespace. Additionally, importing urllib and calling help on it will load parse but not the other modules into the namespace. Jupyter may be loading help proactively causing it to load the parse module only. I don't have IPython/conda/Jupyter installed so can't help to test.