reloading modules (was Re: [Zope3-dev] Re: Google SoC Project)

[Jim Fulton]

Shane Hathaway wrote:
...
> 2) Make reloadable code fundamentally different. 

Yes.

 > If module X is
> supposed to be reloadable, and X creates a module-level global variable 
> Y, and module Z imports Y, then Y needs to be decorated in such a way 
> that Z's view of Y can change automatically when X is reloaded.

A variation on this theme is cause reload to update anything that
can be exported (rather than replacing it).  Of course, this means that
you couldn't export immutable objects, or oreload shouldn't be allowed
to provide a new value for an immutable variable.

Our work on persistent modules should shed some light on this.

I also think there is a real opportunity in allowing reload to fail.
That is, it should be possible for reload to visibly fail so the user
knows that they have to restart.  Then we only reload when we *know*
we can make changes safely and fail otherwise.  For example, in the common
case of updating a class, we can update the class in place.  If there
aren't any other changes, then we know the reload is safe.

> This second approach has subtle limitations, though.  What if Y has the 
> value 10 and Z defines a global variable A whose value is (Y**2)?  The 
> value of A might need to change when Y changes, but how can we arrange 
> for that to happen without making a mess of the code?  I doubt there's 
> any reasonable general solution.

Sure, but realize that this isn't unique to reload.  A client needs to
know if something is idempotent or not.  It should not cache the result of
a non-idempotent operation.

> Even more subtle is what happens when a reloadable module holds a 
> registry of things imported from other modules.  When the module is 
> reloaded, should the registry get cleared?  Zope 2's refresh says the 
> registry should be cleared, but in practice, this confuses everyone.

It causes pain and suffering.  :)

 > To solve this, I think reloadable modules need to have a special global
 > namespace.  Everything in the global namespace, as well as everything
 > reachable from the global namespace, must be explicit about what happens
 > at the time another module imports it or the module is reloaded.  I
 > think this could make a refresh mechanism like the one in Zope 2
 > reliable.  It has a lot of similarity with persistent modules, but it
 > might be simpler.  I haven't thought it all the way through.  The idea
 > came to me about halfway through this post. :-)

Here's an idea: When we do a new-improved reload, we:

1. Reevaluate the code in the pyc, getting the original dictionary.

2. Recompile and evaluate the code without writing a new pyc.

3. Compare the old and new dictionaries to find changes.  If we
    don't know how to compare 2 items, we assume a change.  Note
    removing a variable is considered an unsafe change.  Adding a
    variable is assumed to be a safe change as long as a variable of
    the same name hasn't been added to the module dynamically.

4. We consider whether each of the changes is safe.  If any change
    is unsafe, we raise and error, aborting the reload.  A change is
    safe if the original and new variables are of the same type, the
    values are mutable and if we know how to update the old value
    based on the new value.  In addition, for a change to be safe,
    the original value and the value currently in the module must be
    the same and have the same value.  That is, it can't have been
    changed dynamically.

5. We apply the changes and write a new pyc.

This boils down to merging differences at the Python level.
We fail if we don't know how to apply the diff. At that point,
the user knows they need to restart to get the change.

Hm. This feels kind of workable.  It might even make a good PEP
for a "safe reload".

What do you think?

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org