[ZODB-Dev] Nested OOBTrees?

[Greg Ward]

Hi all --

we're having some strange things going on in our database.  The
background is that a broken web spider (Teleport Pro) recently went on a
rampage on our web site, ignoring session cookies and /robots.txt; the
result is that the number of active sessions in our live database
exploded from its normal ~400-500 range to >7000.

Our database (the relevant area, at least) looks roughly like this:

  root['session_manager'] : MXSessionManager
    .sessions             : OOBTrees { session_id:string -> session:MXSession }

(In English: one of our top-level objects right below the root is the
session manager, an instance of MXSessionManager.  That instance as an
attribute 'sessions', which is an OOBTree mapping session ID strings to
instances of MXSession.)

Normally, root['session_manager'].sessions -- which henceforth I'll call
'sessions' -- has 400 or 500 sessions in it; we clean up old sessions
nightly, so it doesn't grow too large.  The other night, it exploded to
>7000 because of this idiotic web spider (and, err, our lack of defences
against such spiders -- oops).

Now, the sessions OOBTree is acting strangely:

>> len(sessions)                  # OK (we have cleaned up some of the 
3293                              # 7000 bogus sessions)
>> len(sessions.keys())           # OK
3293
>> list(sessions.keys())          # huh?!?
[]

(The values() and items() methods act similarly.)

But I can lookup values from the OOBTree:

>> sid = '12.229.211.187:20020204:064905:****************'
>> sessions[sid]
<MXSession at 000000000013f27c: session 12.229.211.187:20020204:064905:**************** (no user)>

(Session IDs censored because they're sensitive -- the censored bits are
just 16 random hex digits.)

And I can find out at least two session IDs from the OOBTree:

>> sessions.minKey()
'12.229.211.187:20020204:064905:****************'
>> sessions.maxKey()
'80.18.126.195:20020202:044731:****************'

So right now, it just appears that keys(), values(), and items() are
broken.  But if we dig a bit deeper, things get weirder:

>> from ZODB.referencesf import referencesf
>> (pkl, x) = storage.load(sessions._p_oid, None)
>> oids = referencesf(pkl)

With me so far?  You would expect all of the objects referenced from an
OOBTree to be OOBuckets, wouldn't you?  (Well, *I* would, not from any
deep understanding of the design, but simply because that's all I've
ever seen.  But then I've never seen an OOBTree with thousands of
objects before.)  But it isn't that way:

>> print connection[oids[0]]
<OOBTree object at 0x814f4b8>
>> print connection[oids[1]]
<OOBTree object at 0x838c790>
>> print connection[oids[2]]
OOBucket([])

Wot the ... ?!?

So, two questions:

  * are OOBTrees "allowed" to include other OOBTrees?  or are they only
    supposed to reference OOBuckets?

  * in either case, what's wrong with keys()/values()/items() on this
    OOBTree?

Thanks --

        Greg