[ZODB-Dev] ZODB tools for possible contribution

[Greg Ward]

Hi all --

in the course of using and learning about ZODB and ZEO over the past
year or so, we have written a handful of useful and nifty tools that I
think could be useful to everyone using ZODB.  Andrew thinks so too,
which is why you'll find out-of-date versions of some of them in the
contrib/ directory of his SourceForge "zodb' project.  Perhaps the right
thing to do is add these to StandaloneZODB... assuming, that is, that
StandaloneZODB is poised to take over from Andrew's SF project.  Jeremy,
Andrew, any opinions on that?

Anyways, here are the tools we have:

class ObjectDatabase:
  * wrapper around the ZODB database/storage/connection troika
  * "owner" of the database root object (which for us is a
    PersistentMapping of a handful of one-below-the-root-objects,
    which we refer to as "database roots")
  * knows how to open a ZODB in various formats (currently only
    FileStorage and ClientStorage, but adding more storages just
    requires adding more '_init_foo()' methods)
  * the storage type and location (eg. filename or (host, port)) is
    determined by an environment variable, because we have lots of
    little scripts that access the database, and we don't want to
    have to write the same command-line parsing code for every
    one of them
  * keeps a list of database roots and the classes that implement them,
    so if you ask for (eg.) the "user_db" database root, it digs up
    the UserDatabase instance from the dictionary or instantiates
    UserDatabase, puts the new root in the database, and returns it
  * to use it, you subclass and define the _database_roots class
    attribute -- list of (root_name, root_class) pairs

class PersistentList:
  * the obvious analog to PersistentMapping -- it's just a list-like
    object that sets _p_changed when the list is updated
  * already included in Andrew's zodb project, just needs to be
    added to the zope.org CVS

isinstance(), issubclass()
  * implementations of these built-ins that work with ExtensionClass
    with Python < 2.1
  * these will probably be released with Oscar, a type-checking system
    I wrote last winter, one of these weeks

pack_zodb script:
  * open a FileStorage and pack it
  * somewhere, we have code that packs a ClientStorage synchronously --
    ie. it idle-loops until the ".old" file appears, at which point
    the pack is presumably completed.  This could/should be folded
    into this script.

zodb_iterate module/script:
  * provides an interface to iterate over every OID in a ZODB, calling
    a callback function for each one that has an object behind it
  * also provides some functions for manipulating OIDs: pack/unpack/
    format/parse (these probably belong elsewhere, but they live
    in zodb_iterate for now)
  * when run as a script, iterates over a database and prints the
    OID and repr() of every object, in order by OID

zodb_census script:
  * uses zodb_iterate to count the number of instances of every class
    in the database
  * has been posted to this list before

zodb_index script/module:
  * tool to build/query low-level indeces of a ZODB
  * current builds 3:
    - OID -> list of OIDs referenced by it
    - OID -> list of OIDs that reference it
    - class_name -> list of OIDs of instances of that class
  * can do simple reference queries ("show me the list of all objects
    that reference object 0000000000040fa1") or recursive ("show me
    the list of all objects referenced by 000000000040fa1, and all
    objects referenced from them, etc. ad infinitum")
  * eg. to list all objects reachable from the root:
      zodb_index.py -fR 0
    (-f = forward refs, -R = recursive, 0 = OID of the root)
  * can be used as a module -- provides an API to open and query the
    indeces
  * indeces are stored in a separate Berkeley DB file (which can get
    big -- our 45 MB database has a 20 MB index)

Do any of these tools sound useful to you?  Do you think they should be
part of StandaloneZODB?  Turning them into something publicly releasable
would be several days of work; if there's interest, I'd like to get
these tools out into the wider world.

        Greg
-- 
Greg Ward - software developer                gward@mems-exchange.org
MEMS Exchange                            http://www.mems-exchange.org