[Zope] Re: CST 0.9 Problems under load?

Chris McDonough chrism@zope.com
Sun, 17 Feb 2002 17:27:33 -0500


Damn.  I wish I had an easy answer but I don't.

Terry Kerr came up with an important bugfix for CST 0.9 lately, and I'm sure
if I dug in to my inbox I could find a few others that folks have submitted.
These may or may not solve your particular issues.  CST 0.9 seems to have
some fundamental problems with supporting load that are proving very
difficult to pin down because they are particularly subtle and only show up
in situations that are irreplicable.

I'm afraid for now that I'm going to have to suggest that you either wait
for CST 1.0 (which has not ETA as of now) or try the Zope 2.5 Sessioning
machinery.

Sorry,

- C

----- Original Message -----
From: "Tim Hicks" <tim@sitefusion.co.uk>
To: <zope@zope.org>
Cc: "Chris McDonough" <chrism@zope.com>; "Paul Browning"
<paul.browning@bristol.ac.uk>
Sent: Sunday, February 17, 2002 1:28 PM
Subject: CST 0.9 Problems under load?


> Zopers and Chris,
>
> I have written a little zope app that allows people to search an oracle
> database thru-the-web.  Relevant details:
>
> Zope 2.3.3
> Win NT
> DC Oracle 1
> CST 0.9 - used in just about every request as it provided shopping basket
> functionality
> Session timeout set to 20 mins
> Using ZServer (i.e. no apache/squid in front)
> P2 450MHz
> Can't remember how much RAM, but we didn't use it all.
> [From Analog]
> Day:  reqs:  pages:
> Wed: 133586:  2458:
> Thu:  48310:   831:
> Fri:  24890:   438:
>
> We launched the site last Wednesday (10am), and the little box it was
> running on was a bit overwhelmed.  We started to see KeyErrors relating to
> the session data manager (more on that below), so backed off to an earlier
> search app that had no CST component.  CPU had been running at 100% almost
> solidly for about half an hour, but we still had half our RAM free.  At
this
> stage, we weren't logging the errors.
> After the load died down a bit (the first hour was far and away the
worst),
> we switched back to the CST'ified app.  Server load was lower, and
although
> we still weren't logging, we got the impression that the errors had gone.
> Sadly, the KeyErrors were spotted again.  At this point, I setup
> standard_error_message to email me with the REQUEST and traceback whenever
> it caught a KeyError.  That began at 5pm Wednesday.  From then until 1pm
> Friday, I received 174 error messages of the following form:
>
> -----
> <snipped REQUEST>
> Traceback (innermost last):
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in
> publish
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in
> mapply
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in
> call_object
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in
> __call__
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line
> 540, in __call__
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in
> __call__
>     (Object: standard_html_header)
>   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line
> 540, in __call__
>     (Object: standard_html_header)
>   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line
339,
> in eval
>     (Object: sdm.getSessionData().has_key('your_results'))
>     (Info: sdm)
>   File <string>, line 0, in ?
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> er.py, line 264, in getSessionData
>     (Object: sdm)
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> er.py, line 375, in _createSessionDataObject
>     (Object: sdm)
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
> iner.py, line 191, in set
>     (Object: /accom2/search/sdm)
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
> iner.py, line 344, in __setitem__
>     (Object: /accom2/search/sdm)
> KeyError: 64396498A0K2Wz9uuoA
> -----
> As far as I can tell, the only difference between each of the 174 errors
is
> a different key value (corresponding to different sessions I presume).
>
> I also received 4 of this sort:
>
> -----
> Traceback (innermost last):
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in
> publish
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in
> mapply
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in
> call_object
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in
> __call__
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line
> 540, in __call__
>     (Object: results)
>   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in
> __call__
>     (Object: standard_html_header)
>   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line
> 540, in __call__
>     (Object: standard_html_header)
>   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line
339,
> in eval
>     (Object: sdm.getSessionData().has_key('your_results'))
>     (Info: sdm)
>   File <string>, line 0, in ?
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> er.py, line 258, in getSessionData
>     (Object: sdm)
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> er.py, line 390, in _getSessionDataObject
>     (Object: sdm)
>   File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 508, in
> setstate
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
> y, line 156, in load
>     (Object: SessionStorage)
> KeyError:
> -----
> <No key value given!?>
>
> As I said, the relevant REQUEST objects for each of these errors if they
can
> help shed any light on the problems I'm seeing... just ask.
>
> In addition, I also managed to salvage this traceback from about 10.15am
on
> Wednesday, when the CST site was getting hammered.  I only saw it once,
but
> wasn't logging at that stage...
>
> -----
> Error Type: TypeError
> Error Value: function requires exactly 1 argument; 2 given
>
> Traceback (innermost last):
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 223, in
> publish_module
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 187, in
> publish
>   File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 221, in
> zpublisher_exception_hook
>     (Object: Traversable)
>   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 175, in
> publish
>   File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 235, in
commit
>   File C:\PROGRA~1\zope233\lib\python\ZODB\Transaction.py, line 300, in
> commit
>   File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 377, in
> commit
>     (Info: (('BTrees.OOBTree', 'OOBTree'),
> '\000\000\000\000\000\000\000\020', ''))
>   File
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
> y, line 182, in store
>     (Object: SessionStorage)
>   File C:\PROGRA~1\zope233\lib\python\ZODB\ConflictResolution.py, line
177,
> in tryToResolveConflict
>     (Object: SessionStorage)
> TypeError: (see above)
> -----
>
> I don't really no how to try to reproduce these errors short of getting a
> few thousand people to access the site at the same time again.  The CST
site
> is still up and in use under much lighter load.  I haven't received any of
> these errors since Friday lunch time, so it really does seem to be a load
> issue.
>
> I don't have much Zen, so to me it looks like the first error is CST not
> being able to find (losing?) some session information.  The second one is
a
> bit of a mystery to me, and the third looks like it might just be a bug in
> the way the CST code calls ConflictResolution.<some
> class>.tryToResolveConflict()
>
> I'd really appreciate any enlightenment anyone could give about these
> errors, and how to fix them.
>
> cheers
>
> tim
>
>