[ZODB-Dev] Changing namespace - best strategy

[Marius Gedminas]

On Thu, Aug 22, 2013 at 03:40:41PM -0600, Sean Upton wrote:
> Caveat to this below.
> 
> On Mon, Jul 15, 2013 at 7:54 AM, Marius Gedminas <marius at gedmin.as> wrote:
> 
> > On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote:
> > > We need to move a considerable number of persistent objects (~40K) from
> > > one module to another, and we were wondering what is the best strategy
> > > to follow.
> > ...
> > > Has anyone ever done anything like this? Which approach have you
> > > followed? Any suggestions?
> >
> > 1. Make sure all the classes are still importable from the old location
> >    ('from newmodule import MyPersistentSomething # BBB')
> >
> > That is sufficient to make it work.  If you also want to eradicate all
> > references to the old module name from your ZODB (e.g. because you'd
> > like to remove the BBB import), then proceed to step 2:
> >
> > 2. Write a script that loads every instance of this class and does a
> >
> >    obj._p_activate()  # actually not sure this is required, but won't hurt
> >    obj._p_changed = True
> >
> > and then commit the transaction.  Do the commit multiple times, after
> > each batch of several hundred objects, to avoid excessive memory usage.
> > Also be sure to handle conflict errors and retry that batch if you're
> > running this script on the live system.  Finding all instances is left
> > as an exercise for the reader (sometimes findObjectsProviding() helps,
> > if you use nested containers everywhere; sometimes application-specific
> > logic works best; sometimes you end up having to use ZODB iterators to
> > loop through every single object in the DB -- I believe zodbupdate does
> > that.)
> >
> 
> This works only partially, AFAICT.  It will update the stored class name of
> a persistent object.  It will not seek out and change the class name of the
> object in a reference.  Your step 3 (removing the BBB code) can break
> things (broken objects) as a result of this.  Fixing persistent objects is
> half the battle if you get broken references to them.

Ouch, you're absolutely right.

> You can verify that this is incomplete by creating a PersistentMapping of
> some simple objects, perform the trick above, commit, then pack.  You will
> still have the old BBB classname stored in the database in the references,
> unless you do a _p_changed=1 on the mapping containing/referencing the
> items as well (then commit).
> 
> I am not sure how to walk/iterate all oids for all transactions to get all
> possible referencing objects (I assume this is storage-specific, maybe
> building a reference map like Lawrence Rowe has done [1]).  This might be
> necessary to update the referencing objects?

ZODB has an iterator API that allows you to access all the objects.

I think SchoolTool used to have an evolution script that did precisely
this (walk all the objects, set _p_changed to force a write) to solve
this problem.

And then there's https://pypi.python.org/pypi/zodbupdate which sounds
exactly like the tool you want for this.

> [1] http://plone.org/documentation/kb/debug-zodb-bloat/inspectZodbUtils.py

Marius Gedminas
-- 
Killing gnome-session is likely to free substantial amounts of memory, but the
user's gratitude may be surprisingly limited.
                -- Jonathan Corbet in a LWN article about the OOM killer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20130823/b29dfdd3/attachment.sig>