[Zope-dev] Session Errors

Chris McDonough chrism@zope.com
14 Mar 2003 10:30:06 -0500


OK, thanks John.  Let's try one more thing... currently the mounted
database used to store the session data uses a connection that ignores
read conflicts.  This is known to be bad because the machinery which
deals with keeping the sessioning index data will also ignore read
conflicts, which may create inconcstencies between two data structures
(BTrees) that need to be kept in sync.

Here's a patch to lib/python/Products/TemporaryFolder/TemporaryFolder.py
that reenables read conflict generation on the database.

Index: TemporaryFolder.py
===================================================================
RCS file:
/cvs-repository/Zope/lib/python/Products/TemporaryFolder/TemporaryFolder.py,v
retrieving revision 1.7
diff -r1.7 TemporaryFolder.py
72c72
<         db.klass = LowConflictConnection
---
>         #db.klass = LowConflictConnection

You may see many more conflicts with this running.  But maybe the data
structures will not become desynchronized.

Another problem, still unexplained, experienced by Andrew Athan, is that
if a reference is made to a session data object from within the standard
error message, somehow things get screwy under high load.  If you're
doing the same, please let me know.

- C
 


On Fri, 2003-03-14 at 06:48, John Eikenberry wrote:
> Chris McDonough wrote:
> 
> > OK, thanks John.
> 
> Thank you for helping.
>  
> > I hate to ask this (I should have done this to start with), but would
> > you be willing to use the following patch --against the original file,
> > not your recently patched version-- and try again?  I only checked one
> > of the two BTrees that might be at the heart of the problem with the
> > first patch, this patch checks the second as well.
> 
> Put the patch in place and have a couple errors already. Doesn't look like
> its going to be much help though. Just a couple repetitions of this:
> 
> ------
> 2003-03-14T03:35:53 PROBLEM(100) Transience KeyError raised in get,
> checking BTrees
> ------
> 2003-03-14T03:35:53 PROBLEM(100) Transience BTree check for data succeeded
> ------
> 2003-03-14T03:35:53 PROBLEM(100) Transience BTree check for index succeeded
> 
> 
> 
> > - C
> > 
> > 
> > On Thu, 2003-03-13 at 18:18, John Eikenberry wrote:
> > > 
> > > Patch applied and the first results are in... so far its a lot of these:
> > > 
> > > 
> > > 2003-03-13T15:18:07 PROBLEM(100) Transience KeyError raised in get,
> > > checking _data BTree
> > > ------
> > > 2003-03-13T15:18:07 PROBLEM(100) Transience BTree check succeeded
> > > 
> > > 
> > > Chris McDonough wrote:
> > > 
> > > > Hi John,
> > > > 
> > > > Can you apply the attached diff to your Transience.py file and run with
> > > > it in place for a couple of days?  It will not fix the problem (the
> > > > symptoms will remain) but it should print some diagnostic information to
> > > > the Zope event log (the STUPID_LOG_FILE, hopefully you've got that
> > > > going) that will help us track down what this might be.
> > > > 
> > > > Once you notice it happen, send the relevant parts of your logfile to me
> > > > and I will see if I can analyze it.
> > > > 
> > > > - C
> > > > 
> > > > 
> > > > 
> > > > 
> > > > On Thu, 2003-03-13 at 15:19, John Eikenberry wrote:
> > > > > 
> > > > > Sorry, its Zope 2.6.1.
> > > > > 
> > > > > Chris McDonough wrote:
> > > > > 
> > > > > > John,
> > > > > > 
> > > > > > Which Zope 2.6?  Zope 2.6.1?  Here's what line 807 of the current
> > > > > > Transience.py looks like:
> > > > > > 
> > > > > >                 v = self._data[b].get(k, notfound)
> > > > > > 
> > > > > > Does yours look like that?
> > > > > 
> > > > > Yes.
> > > > > 
> > > > > > What is the value of the __version__ variable at the top of the
> > > > > > Transience.py file?
> > > > > 
> > > > > __version__='$Revision: 1.28.6.4 $'[11:-2]
> > > > > 
> > > > >  
> > > > > > On Thu, 2003-03-13 at 07:11, John Eikenberry wrote:
> > > > > > > Since upgrading to Zope-2.6 we've been getting KeyErrors when using
> > > > > > > Sessions. They seem to happen more now that we've started using
> > > > > > > hasSessionData(), but I'm pretty sure they happened prior to that.
> > > > > > > 
> > > > > > > Anyways, here are the 2 related tracebacks. Has anyone else seen these?
> > > > > > > 
> > > > > > > Traceback #1 occurs most frequently. The KeyError's value is an unprintable
> > > > > > > string of non-ascii characters.
> > > > > > > 
> > > > > > > * Module ZPublisher.Publish, line 150, in publish_module
> > > > > > > * Module ZPublisher.Publish, line 114, in publish
> > > > > > > * Module The application server.App.startup, line 182, in
> > > > > > >   zpublisher_exception_hook
> > > > > > > * Module ZPublisher.Publish, line 98, in publish
> > > > > > > * Module ZPublisher.mapply, line 88, in mapply
> > > > > > > * Module ZPublisher.Publish, line 39, in call_object
> > > > > > > * Module App.special_dtml, line 61, in __call__
> > > > > > > * Module DocumentTemplate.DT_String, line 474, in __call__
> > > > > > > * Module Products.Transience.Transience, line 342, in nudge
> > > > > > > * Module Products.Transience.Transience, line 467, in _getCurrentBucket
> > > > > > > * Module Products.TemporaryFolder.LowConflictConnection, line 34, in
> > > > > > >   setstate
> > > > > > > * Module Products.TemporaryFolder.TemporaryStorage, line 94, in load
> > > > > > > KeyError:
> > > > > > > 
> > > > > > > Traceback #2 happens less frequently, though today it seemed like it was
> > > > > > > trying to catch up (3 of these today).
> > > > > > > 
> > > > > > > * Module ZPublisher.Publish, line 98, in publish
> > > > > > > * Module ZPublisher.mapply, line 88, in mapply
> > > > > > > * Module ZPublisher.Publish, line 39, in call_object
> > > > > > > * Module OFS.DTMLMethod, line 126, in __call__
> > > > > > > * Module DocumentTemplate.DT_String, line 474, in __call__
> > > > > > > * Module Products.DotOrg.Pages.KContent, line 149, in __call__
> > > > > > > * Module Products.DotOrg.Pages.KContent, line 194, in getEditInfo
> > > > > > > * Module Products.DotOrg.Pages.KContent, line 506, in hasSessionData
> > > > > > > * Module Products.Sessions.SessionDataManager, line 101, in hasSessionData
> > > > > > > * Module Products.Sessions.SessionDataManager, line 175, in
> > > > > > >   _hasSessionDataObject
> > > > > > > * Module Products.Transience.Transience, line 838, in has_key
> > > > > > > * Module Products.Transience.Transience, line 807, in get
> > > > > > > 
> > > > > > > KeyError: 1047409860 
> > > > > > > 
> > > > > > > 
> > > > > > > -- 
> > > > > > > 
> > > > > > > John Eikenberry [jae@kavi.com]
> > > > > > > ______________________________________________________________
> > > > > > > "A society that will trade a little liberty for a little order
> > > > > > >  will deserve neither and lose both."
> > > > > > >                                           --B. Franklin
> > > > > > > 
> > > > > > > _______________________________________________
> > > > > > > Zope-Dev maillist  -  Zope-Dev@zope.org
> > > > > > > http://mail.zope.org/mailman/listinfo/zope-dev
> > > > > > > **  No cross posts or HTML encoding!  **
> > > > > > > (Related lists - 
> > > > > > >  http://mail.zope.org/mailman/listinfo/zope-announce
> > > > > > >  http://mail.zope.org/mailman/listinfo/zope )
> > > > > > 
> > > > > > 
> > > > > 
> > > > > -- 
> > > > > 
> > > > > John Eikenberry [jae@kavi.com]
> > > > > ______________________________________________________________
> > > > > "A society that will trade a little liberty for a little order
> > > > >  will deserve neither and lose both."
> > > > >                                           --B. Franklin
> > > > > 
> > > > > _______________________________________________
> > > > > Zope-Dev maillist  -  Zope-Dev@zope.org
> > > > > http://mail.zope.org/mailman/listinfo/zope-dev
> > > > > **  No cross posts or HTML encoding!  **
> > > > > (Related lists - 
> > > > >  http://mail.zope.org/mailman/listinfo/zope-announce
> > > > >  http://mail.zope.org/mailman/listinfo/zope )
> > > > 
> > > 
> > > > ? btreecheck.diff
> > > > ? kedaipatch
> > > > Index: Transience.py
> > > > ===================================================================
> > > > RCS file: /cvs-repository/Zope/lib/python/Products/Transience/Transience.py,v
> > > > retrieving revision 1.28.6.4
> > > > diff -r1.28.6.4 Transience.py
> > > > 34a35
> > > > > from BTrees.check import check, display
> > > > 45a47
> > > > > from cStringIO import StringIO
> > > > 807c809,830
> > > > <                 v = self._data[b].get(k, notfound)
> > > > ---
> > > > >                 try:
> > > > >                     v = self._data[b].get(k, notfound)
> > > > >                 except KeyError:
> > > > >                     LOG('Transience', WARNING,
> > > > >                         'KeyError raised in get, checking _data BTree')
> > > > >                     try:
> > > > >                         check(self._data)
> > > > >                         LOG('Transience', WARNING,'BTree check succeeded')
> > > > >                     except:
> > > > >                         # BTree check failed
> > > > >                         oldstdout = sys.stdout
> > > > >                         tmp = StringIO()
> > > > >                         try:
> > > > >                             sys.stdout = tmp
> > > > >                             display(self._data)
> > > > >                             tmp.seek(0)
> > > > >                             LOG('Transience', WARNING,'BTree check failed',
> > > > >                                 tmp.read(), error=sys.exc_info())
> > > > >                             tmp.close()
> > > > >                         finally:
> > > > >                             sys.stdout = oldstdout
> > > > >                     raise
> > > 
> > > 
> > > -- 
> > > 
> > > John Eikenberry [jae@kavi.com]
> > > ______________________________________________________________
> > > "A society that will trade a little liberty for a little order
> > >  will deserve neither and lose both."
> > >                                           --B. Franklin
> > > 
> > > _______________________________________________
> > > Zope-Dev maillist  -  Zope-Dev@zope.org
> > > http://mail.zope.org/mailman/listinfo/zope-dev
> > > **  No cross posts or HTML encoding!  **
> > > (Related lists - 
> > >  http://mail.zope.org/mailman/listinfo/zope-announce
> > >  http://mail.zope.org/mailman/listinfo/zope )
> > 
> 
> > Index: Transience.py
> > ===================================================================
> > RCS file: /cvs-repository/Zope/lib/python/Products/Transience/Transience.py,v
> > retrieving revision 1.28.6.4
> > diff -r1.28.6.4 Transience.py
> > 807c807,814
> > <                 v = self._data[b].get(k, notfound)
> > ---
> > >                 try:
> > >                     v = self._data[b].get(k, notfound)
> > >                 except KeyError:
> > >                     LOG('Transience', WARNING,
> > >                         'KeyError raised in get, checking BTrees')
> > >                     check_btree(self._data, 'data')
> > >                     check_btree(index, 'index')
> > >                     raise
> > 954a962,981
> > > 
> > > def check_btree(btree, name):
> > >     from BTrees.check import check, display
> > >     from cStringIO import StringIO
> > >     try:
> > >         check(btree)
> > >         LOG('Transience', WARNING,'BTree check for %s succeeded' % name)
> > >     except:
> > >         # BTree check failed
> > >         oldstdout = sys.stdout
> > >         tmp = StringIO()
> > >         try:
> > >             sys.stdout = tmp
> > >             display(btree)
> > >             tmp.seek(0)
> > >             LOG('Transience', WARNING, 'BTree check for %s failed' % name,
> > >                 tmp.read(), error=sys.exc_info())
> > >             tmp.close()
> > >         finally:
> > >             sys.stdout = oldstdout
> 
> 
> -- 
> 
> John Eikenberry [jae@kavi.com]
> ______________________________________________________________
> "A society that will trade a little liberty for a little order
>  will deserve neither and lose both."
>                                           --B. Franklin
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev@zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope )