[Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)

[Just van Rossum]

Guido van Rossum wrote:

> > Playing tricks with the contents of pkg.__path__ apparently has its
> > uses,
> 
> It's the *defined* API for adding to the set of locations from which
> submodules and subpackages of a package are to be loaded.  You can't
> just call that "playing tricks."

pkg.__path__ is a package-local equivalent of sys.path. But if a package
didn't *originate* from sys.path, it makes little sense to demand
pkg.__path__ be a list of strings.

But: it could then just as well be an empty list. With my patch I think
frozen package imports could even be fixed to do that. Hm, that would be
real easy in fact. Will try this out.

> Does the metahook also apply to submodules (or subpackage) of
> packages?

Of course it does. An importer must be able to handle submodules of
packages it loaded, if it wants. Frozen import works that way: in fact
you can see the frozen import mechanism as an imaginary importer on
sys.meta_path. sys.meta_path is simply traversed before
sys.path/pkg.__path__, always.

A big difference between my hooks and the classic
imp.find_module/load_module functions is that my hooks *always* get
called with the fully qualified module name (and have to look up
sys.path or pkg.__path__ by themselves if they need it), and
imp.find_module must be called with a (sub)name and a path argument. The
latter is fine for of sys.path-based imports but is a pain for any
other, as then you somehow need to reconstruct the parent name (which
actually led to freeze (ab)using __path__ to communicate the parent
name).

> I'd expect not.  (I haven't had the time to review your latest
> patches yet, in part due to this thread. :-)

Oh, this aspect hasn't changed since since I first posted it on sf, but
I think you spent more time in zipimport.c than in import.c ;-)

> Surely you shouldn't be looking for builtin submodules of a package.

Why not? (Builtin submodules won't work anyway at the moment, but for
other reasons.)

> >     Depending on the kind of importer, there are different
> >     levels of freedom of what you can use as pkg.__path__.
> >     
> >     Importer object on sys.meta_path:
> >         it can use anything it pleases (even None), as long
> >         as a __path__ variable is set.
> 
> But I can imagine that some metahooks would like to look inside the
> __path__ list for more hints on where to find the submodules of the
> package.  Of course that's up to the metahook.

Exactly. __path__ can be seen as private to an importer. It just happens
to be a list of strings for the sys.path importer ;-).

> >     Importer object on sys.path:
> >         pkg.__path__ must be a list; it's most logical to use
> >         an importer object as the only item. Could be the same
> >         importer instance that imported the package itself.
> 
> Multiple items would make sense too.

Sure.

> >     A hook on sys.path_hooks:
> >         pkg.__path__ must be a list and its only item should
> >         be a string that the hook can handle itself.
> 
> IMO it should allow multiple strings too!

It sure does, I formulated that clumsily. If no meta_path hook picks up
the import, sys.path or pkg.__path__ traversal does its work as usual.
Apart from the sys.path_hooks hookability of sys.path/pkg.__path__
items, nothing was changed here.

> >     These are just guidelines: a set of hooks could in theory
> >     deliberately set pgk.__path__ up so submodule imports be
> >     handled by an entirely different importer. Not sure how
> >     useful that would be...
> 
> Very useful, for not-yet-imagined cases.

I was just pointing out that it's possible...

Just