[ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

[Chris McDonough]

I had another thought about this.  It's a bit of voodoo, but if Alex is
willing to do it, and it still fails, it rules out the storage as a
culprit.

Alex, do you think you could try to use a different storage than a
TemporaryStorage for the database that backs the temporary folder?  You
can do this by replacing this text in your zope.conf file:

<zodb_db temporary>
    # Temporary storage database (for sessions)
    <temporarystorage>
      name temporary storage for sessioning
    </temporarystorage>
    mount-point /temp_folder
    container-class Products.TemporaryFolder.TemporaryContainer
</zodb_db>

With this text:

<zodb_db temporary>
    # Temporary storage database (for sessions)
    <filestorage>
      path $INSTANCE/var/Sessions.fs
    </filestorage>
    mount-point /temp_folder
    container-class Products.TemporaryFolder.TemporaryContainer
</zodb_db>

This will use a FileStorage to back the sessioning database instead of a
TemporaryStorage.  This database file will grow and grow and grow,
unlike a TemporaryStorage, which does garbage collection of unused nodes
and stores its data in RAM.  But temporarily, I think it would be fine
and if you continue to see the error, it would assure me that
TemporaryStorage is doing its job properly.

- C

On Wed, 2004-03-03 at 15:15, Chris McDonough wrote:
> On Wed, 2004-03-03 at 12:44, Jeremy Hylton wrote:
> > >         for key in list(self._data.keys(None, max_ts)):
> > >             assert(key <= max_ts)
> > >             STRICT and _assert(self._data.has_key(key))
> > >             for v in self._data[key].values():
> > >                 to_notify.append(v)
> > >             del self._data[key]
> > 
> > I don't have much context for this question.  It's definitely the case
> > that a corrupt BTree there are keys you can reach using keys(), which
> > follows the bucket next pointers, that can't reach using a lookup, which
> > follows child pointers down through the interior nodes.
> 
> Right, I figured as much.
> 
> > If you could call the check functions on the BTrees in question.  That's
> > object._check() to check C internals and BTrees.check.check() to check
> > value based consistency.
> 
> I'm hoping Alex will do this for us as I haven't been able to make the
> error occur in isolation.
> 
> > So how is the BTree is question used?  If the test is failing here, it
> > seems most likely that the BTree was corrupted by a write somewhere
> > else.
> 
> The IOBTree in question (_data) is a mapping from an integer to an
> OOBTree.  The integer key represents a time period value, the OOBTree
> value is a mapping of session ids to session objects which were created
> within this time period.  The _data BTree is of course written to
> elsewhere in the code
> (http://cvs.zope.org/*checkout*/Products/Transience/Transience.py?rev=1.32.12.2.2.2&content-type=text/plain)
> 
> If anybody (there's that *anybody* again, can you feel your ears burning
> Jeremy?) can grep that code for _data and see if there's someplace I'm
> doing something suspicious with that BTree, I would be grateful.  The
> last time I had a problem like this it turned out I was iterating over
> the results of a BTree's keys() method (which returns an iterator) and
> deleting those keys out of the BTree in the loop body, which caused the
> issue.  But I've been careful to not do that, and as far as I can tell
> there's nothing about the code that does anything terribly dumb.  This
> is corroborated by unit tests for that code, and the fact that the error
> doesn't occur under "light" load; it appears just to happen in
> situations where the transience code is executed in parallel by  more
> than one thread (thus leading me to believe it may have something to do
> with conflict resolution).
> 
> - C
> 
> 
> 
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
> 
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev