[ZODB-Dev] Re: Invalidating caches for module refresh doesn't seem to work

[Jim Fulton]

On Jan 30, 2007, at 11:12 AM, Philipp von Weitershausen wrote:

> Jim Fulton wrote:
>> On Jan 29, 2007, at 5:46 PM, Philipp von Weitershausen wrote:
>>> After refreshing a product, Zope 2 uses the following stanza in  
>>> App.RefreshFuncs.autoRefresh() to let the ZODB know that it  
>>> should invalidate its pickle caches:
>>>
>>>         ... refresh products
>>>         from ZODB import Connection
>>>         Connection.resetCaches()
>>>         transaction.commit()
>>>         jar._resetCache()
>>>         transaction.begin()
>> That is really weird code.  It looks very brittle.
>>> That way, persistent objects will use the newly reimported  
>>> classes instead of continuing to use the old, no longer current  
>>> classes.
>> I suppose that was the intent.
>>> For grok I tried to implement the same mechanism and copied that  
>>> code almost verbatimly, only to find out it doesn't seem to work.  
>>> Persistent objects will still be instances of the classes that  
>>> should have been thrown away during the re-import of modules.
>>> Testing product refresh with both Zope 2.9 and 2.10 produces  
>>> errors for me while Zope 2.8 works, which leads me to the  
>>> assumption that a cache invalidation bug was introduced after  
>>> ZODB 3.4. Is anybody else seeing this?
>> The way that connections were managed changes quite a bit in 3.4.
>>> I'm a bit surprised nobody else has complained about this so far...
>> Maybe people finally stopped using product-refresh because it  
>> doesn't work
>> reliably.
>
> What do I have to do to make it work reliably? I'd like to avoid  
> having to restart all of Zope. (Note that this is grok on Zope3,  
> not necessarily Zope 2).

There is nothing you can do. Bonus: Zope 3 makes it even harder for  
reasons that have nothing to do with ZODB.

>>> Help with tracking this down would be greatly appreciated.
>> Shane originally wrote this.  Maybe he could help out. I'm not  
>> interested myself. In fact, I'l be happy to see this code get   
>> ripped out.  If it stays in it needs a doctest to prevent future  
>> regression and to explain how to use it, including what it's  
>> semantics and limitations are.
>
> So, are there any alternatives that let me refresh modules w/o  
> server restarts AND make the ZODB aware of this? Would closing and  
> reopening the database altogether be an option? I would assume it  
> would leave the connections that other threads have to the database  
> in a funny state, but I don't understand a whole lot about ZODB  
> internals so I'm just guessing.

The reliability problem isn't with ZODB.  The cache-refresh thing can  
be made to "work" there, if anyone cares.  (Honestly, I'd rather see  
the feature go away to make ZODB a little bit simpler.)

The problem is with module globals that are outside the scope of  
ZODB.  The problem with product refresh is that it seems to work much  
of the time.  When it doesn't work, people get really weird bugs that  
they waste a lot of time chasing, forgetting that module reload is  
involved.

Python modules were not designed to be reloadable.  If you want to  
change that, I suggest taking that up on the Python 3000 list. (Maybe  
someone already has. Regrettably, I don't have time to watch that.)  
Period.  ZODB appears to provide a clever way to get around some of  
the inherent module-reload problems, but it only provides a partial  
solution.

My advice is to focus on making restarts faster.  But that discussion  
should happen elsewhere, not on the ZODB-dev list.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org