[ZCM] [ZC] 413/ 4 Comment "Zope hangs while updating multiple ParsedXML objects"

[Collector: Zope Bugs, Features, and Patches ...]

Issue #413 Update (Comment) "Zope hangs while updating multiple ParsedXML objects"
 Status Pending, Zope/bug medium
To followup, visit:
  http://collector.zope.org/Zope/413

==============================================================
= Comment - Entry #4 by allison on Jun 5, 2002 4:22 pm

Still missing the cause, but the workaround I found might help. Adding another 256MB of memory to the 64MB used previously caused the problem to go away.  That suggests that the bug is somewhere in the inteface between Zope and the Linux virtual memory system.  It's unclear whether ParsedXML has anything to do with the problem at all although its insertion/deletion pattterns are different than most others (or so I suspect).
________________________________________
= Comment - Entry #3 by allison on Jun 3, 2002 7:04 pm

Another round, this time without the option of doing a global delete and build.  The update goes smoothly for a bit, then slows to a crawl (or slower) with lots of disk traffic.  After a long time (hours) the update for the fist data set completed. The second data set did not finish as my patience ran out after 6 hours....  (Builds with the same code take minutes.)  Poking around, it appears that the python image has grown to 100MB or so even though there is nothing anywhere near that size around.
This particular machine has limited memory (64MB).  The problem seems to arise when swapping is required anbd the overheads of threashing  dominate useful throughput.  Looks like the ParsedXML + Python somehow is leaking memory.
________________________________________
= Comment - Entry #2 by allison on Jun 1, 2002 3:01 pm

I removed the need to delete by doing a single global delete 
of the folder tree so everything became a build.  With that 
strategy, I was able to compete an upload/update.  The external method doing the load completed (logged to the Zope terminal) 
a full minute to a minute and a half before the "thank-you-very-much" response page was handed back.  As the
delete/insert strategy would have created a queue twice as long, the problem may lie with queue management and/or patience of the operator, or something related to the performance issues 
seen and cited in issue #83.
________________________________________
= Request - Entry #1 by allison on Jun 1, 2002 2:15 pm

This one is hard to localize, but it is easy enough to trigger.   I have a set of nested folders with ParsedXML objects in the leaves (and in some of the interior folders as well).  These are loaded (initially) from an external file system using an external procedure.  Update involves walking the folder structure and deleting the old ParsedXML object and adding a new one.  There are order hundreds of such objects, but each is fairly small (1K-15K or so).  The observed behavior is that the updating moves forward for a while, then stops and does something for a long long time.  Sometimes it returns after minutes, but then gets back into the same state.

When this happens, Zope is pretty much unavailable to any user and locked down.

If the processing is upload mostly with only a few (if any) deletions, the processing goes happily to completion and does not tickle this problem.

Given this behavior, I suspect some sort of locking problem where a lock is not being cleared.

There is an open problem related to ParsedXML (issue #83) which crashes Zope which may (or may not) be related.

 
==============================================================