[ZODB-Dev] Tool for exploring raw ZODB data

[Jeff Sasmor]

1. Ignore 'redir', it's specific to what I'm doing 
and you don't need to worry about it. It's just a 
way to have a 'link' within a folder that short-
circuits the traversal process.

2.The whole point of what this code does is to
pull out the object structure without actually
creating the objects. And yes, it does assume that
whatever it finds is an object with a straightforward
dictionary holding the attrs. 

The bizzare-looking stuff you see is part of the 
ZODB's trickery.magic: normally when you 
access ZODB  thru a connection instance it 
uses the unpicker's  persistent_load hook; 
this causes the object to be recreated by the 
use of an import. 

Although this explanation is a little oversimplified,
basically The  ('\x00\x00\x00\x00\x00\x01>\xce',
     ('IndexedCatalog.Catalog', 'Catalog')) format
tells the persistent_load hook how to recreate the objm
where to get the class from: the first part 
(IndexedCatalog.Catalog) is the module, 
The second part (Catalog) is the class.
The odd looking # is the OID. So when the ZODB
and the Connection try to access an object, all the
subobjs can be located within the ZODB with the OID,
and the structure of the obj specified by the module and
class. So the whole shebang can be built.

But I don't want to do that. All I want to do is to get
attributes of an object so I can access it quickly 
with a small memory footprint. Hence, I use 
unpickler.noload to just recreate the structure - simple
attrs are loaded and subobjs appear as tuples with
an OID and class reconstruction info. Loading these
is more tricky as you see.


It's hard for me to tell what's going on in your case
without seeing what you've got in your ZODB, how
it's organized,  and what you're trying to access. 
If it's something in a container/subobject heirarchy 
using dictionaries (like what I'm doing or within 
Zope) then it ought to work. If you're
trying to pull out a subobject of an object that's a 
persistent mapping or a BTree then you'll probably
have to modify the code or do some additional
processing as you seem to have found out. This
was not unexpected - I don't need to be able to
pull those apart so didn't get that deep - it could
certainly be done though.

What I wanted to do was to be able to get the dict
for an object so that I could retrieve the contents
of individual attributes. If the attr is a BTree then
there's probably more work involved. 

Didn't mean to confuse anyone!
#--------------------------------
Jeff Sasmor
jeff@sasmor.com
----- Original Message ----- 
From: "Christian Reis" <kiko@async.com.br>
To: "Jeff Sasmor" <jsasmor@gte.net>
Cc: <zodb-dev@zope.org>
Sent: Friday, February 14, 2003 8:02 PM
Subject: Re: [ZODB-Dev] Tool for exploring raw ZODB data


On Fri, Feb 14, 2003 at 05:56:03PM -0500, Jeff Sasmor wrote:
> "Christian Reis" <kiko@async.com.br> wrote:
> 
> >Looks cool, but I have a question. How are the objects organized into
> >paths, as this seems to be hardcoded into the application. Was it
> >designed for something like Zope uses?
> 
> No, actually it doesn't matter what system you're using; this works
> at too low a level to care. The ZODB is the same no matter
> what's 'above' it. Even in Zope, there's a path from the Root Folder
> thru folders, etc.

I think it *does* matter, or else this would work with my ZODB files :-)
No, honestly, let me understand if there's an assumption being made of
if the ZODB has changed enough to break things here.

(A note; I really don't get what redir_* are at all! My mind must be
toast from my thesis work this week :-/ )

We have:

        obj = getObjectDirect(path,OPath,redir_info)
        if obj:
            for k,v in obj.items():

However, the obj I get back from *my* database isn't a dictionary; it's
a pretty deep tuple. The funny part is that I *am* asking for a mapping
(I'm pulling root['Sequences'] which is a BTree). Hmmm. Does this mean
this code is assuming we're going to pull a PersistentMapping and not a
BTree back? Hmm! Here's my tuple; it's pretty deep, but at the bottom it
contains a set of tuples containing

    (key, oid, module, klass)

(((('Companies',
    ('\x00\x00\x00\x00\x00\x01>\xce',
     ('IndexedCatalog.Catalog', 'Catalog')),
    'SOProducts',
    ('\x00\x00\x00\x00\x00\x01>\xd8',
     ('IndexedCatalog.Catalog', 'Catalog')),

    [ snip bunch of boring objects ]
     
    'SOrders',
    ('\x00\x00\x00\x00\x00\x01>\xda',
     ('IndexedCatalog.Catalog', 'Catalog')),
    'ThirdParties',
    ('\x00\x00\x00\x00\x00\x01>\xdb',
     ('IndexedCatalog.Catalog', 'Catalog'))),),),)

This causes an exception to be raised since (of course) tuples don't
have an items() method. Hmmm; so it seems that getObjectDirect() isn't
pulling out an object, but a tuple that represents what it holds. I
think.

What I don't understand, truly, is *what* this tuple is. Where does this
bizarre format come from?

> So you can see that this tool is actually platform independent.
> Just think of the + as a '.' or a '/'.

I still don't understand how this object traversal should work. Does
this assume that everything from the root object onwards is either an
object or a PersistentMapping? And BTrees don't work as expected?

> It wasn't designed for Zope, but I think if you're developing
> a Zope app it might be cool to use this to see if you have

I'm not; this is hardcode independent ZODB usage :)

> the data in the ZODB that you _think_ you have. Perhaps
> it would be handy when debugging unusual problems. It's
> also instructive if you're trying to understand how this
> layer of the ZODB works.

Yeah, I found it to be *really* cool, but I'm still poring through the
code to get a hang for it.

Take care,
--
Christian Reis, Senior Engineer, Async Open Source, Brazil.
http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL