[ZCM] [ZC] 1350/13 Comment "Zope HEAD (2.8) chokes under high load"

[Collector: Zope Bugs, Features, and Patches ...]

Issue #1350 Update (Comment) "Zope HEAD (2.8) chokes under high load"
 Status Pending, Zope/bug critical
To followup, visit:
  http://collector.zope.org/Zope/1350

==============================================================
= Comment - Entry #13 by dunny on Oct 5, 2004 4:55 am

I also get the segfaults when using TemporaryStorage for transient object container. But not when using FileStorage.

Seems to break in a few different places. (I have yet to collect any details.) 

Also get segfaults when trying with a debug Python (in this case version 2.4a3) and Zope. 

Furthermore it looks like session machinery is somehow dropping session data well before the segfault. TestSessionRig reports of missing session data for bid's that should exist according to its own log.
________________________________________
= Comment - Entry #12 by tim_one on Oct 4, 2004 4:29 pm

1. The first thing to do with a segfault is to build everything in debug mode, both Python and Zope.  Python in particular has lots of C assert()s that vanish in a release build.

2. When I was trying to stitch a current ZODB into the Zope trunk, I got many segfaults at first.  I don't know what caused any of them, because they all went away by themselves, as I added code to explicitly abort transactions that had suffered a failing commit, and fixed other incompatibilities.  Maybe something is left in an insane state if you don't do that.  Since the segfaults went away without trying to track down any of 'em, I never found out!
________________________________________
= Comment - Entry #11 by mcdonc on Oct 4, 2004 4:11 pm

It looks like the segfault is caused by someting that depends largely on some sequence of events happening in a particular order (which happen to be evoked by the session rig tests) as opposed to some overflow, because I just ran the rig test again in the same manner and didn't get a segfault for almost 15 minutes as opposed to about 5 before.
________________________________________
= Edit - Entry #8 by mcdonc on Oct 4, 2004 3:49 pm

 Changes: submitter email, edited transcript
________________________________________
= Resubmit - Entry #7 by mcdonc on Oct 4, 2004 3:48 pm

 Status: Resolved => Pending

Spoke a bit too soon.

Running the session rig against the current trunk in the manner indicated in an earlier message to this issue indeed no longer deadlocks but now provokes a segfault.  For me, the segfault happens after ~ 5 minutes of continued pounding; not immediately.  (Note that it is no longer required that you set up a "/toc" object in the Zope root because the temp folder appears to work now).

Updated "rig files" are in CVS at cvs.zope.org:/cvs-repository within Packages/SessionRig.  If you try to reproduce, please use the files in CVS rather than the files uploaded to this issue.
________________________________________
= Resolve - Entry #6 by mcdonc on Oct 4, 2004 3:44 pm

 Status: Pending => Resolved

The deadlock problem appears to have been solved at some point since this bugreport (maybe since Tim's merge of the 3.3 HEAD?).  Running the test rig reports ancillary data consistency errors ("wrong session data"), but these are unlrelated to the deadlock and may be a problem of the test rig itself.
________________________________________
= Comment - Entry #5 by mcdonc on May 23, 2004 2:40 am


Uploaded:  "SessionRigExtensions.py"
 - http://collector.zope.org/Zope/1350/SessionRigExtensions.py/view

________________________________________
= Comment - Entry #4 by mcdonc on May 23, 2004 2:39 am


Uploaded:  "TestSessionRig.py"
 - http://collector.zope.org/Zope/1350/TestSessionRig.py/view

________________________________________
= Comment - Entry #3 by mcdonc on May 23, 2004 2:39 am


Uploaded:  "sessionrig.zexp"
 - http://collector.zope.org/Zope/1350/sessionrig.zexp/view

________________________________________
= Comment - Entry #2 by mcdonc on May 23, 2004 2:37 am

Note also that the same stresstest completes ok against the 2.7 branch code.
________________________________________
= Request - Entry #1 by mcdonc on May 23, 2004 2:37 am

It seems there is a deadlock problem somewhere in the code on the Zope HEAD.  I noticed this when running a high-stress session test against it.  The symptom is that it "freezes" shortly after maybe 40 or 50 near-simultaneous HTTP requests.  This happens with and without the "error handling in main transaction" changes I just made to ZPublisher/Publish.py and Zope/App/startup.py .  The code was also freshly compiled when I ran the test.

To replicate:

- Download the attached files:

  - TestSessionRig.py -- a runnable script which
    hammers Zope with simultaneous read/write HTTP
    requests in order to test sessioning stability.
    This can be placed anywhere.

  - sessionrig.zexp -- an importable 'zexp' file, place
    this in Zope's import directory.

  - SessionRigExtensions.py -- an 'external method'
    module, which is used by sessionrig.zexp.  It should
    be placed in Zope's 'Extensions' folder.

Set up:

  - import sessionrig.zexp into the root folder using
    the ZMI.

  - Create a "transient object container" named "toc"
    within the root folder using the ZMI "Add.." dropdown.
    Set its "data object timeout value" to 1 minute.

  - Visit the session_data_manager object present in
    Zope's root folder and change it's 'transient object
    container path' to '/toc'.

Run the test:

  - Start Zope.

  - Edit the TestSessionRig.py file and change the 'path'
    global from the default "http://localhost:8085/sessionrig/customerror'
    to something that matches your Zope's HTTP port
    number.

  - run "python TestSessionRig.py"

The test will "crap out" after some short period of time; for me it is about 20 seconds.  The CPU usage is basically nil on the process when it does so, but it seems as if all database connections are consumed, because the ZMI is inaccessible.
==============================================================