[ZODB-Dev] Zope or Zeo hanging up sockets
Matt
matt@virtualspectator.com
Wed, 28 Nov 2001 16:44:21 +1300
I would be happy to help.
First of all, and don't laugh, the following are the external methods
involved :
def makeHQOned(self,REQUEST,RESPONSE):
RESPONSE.setHeader('content-type','text/xml')
if hasattr(self.aq_base,'oned_std'):
oned_txt = getattr(self.aq_base,'oned_std').content
new_oned = change_version(oned_txt)
return new_oned
else:
return "<error/>"
def change_version(oned_txt):
document_node = vs_dom.ReadXMLFromString(oned_txt)
vs_xpath.modifyElementAttributes(document_node,'//interval/@value','40')
new_text = vs_dom.retPrettyPrint(document_node)
return new_text
Basically, all it does is make sure there is an object called oned_std in
the current container object that the method HQOned is called on. This
object is a simple Product of mine where content is just a python string
that holds some xml. This method calls change_version which 1) creates a
dom document of the 4Suite kind, 2) uses xpath to find the node I want to
change, and 3) print the result back into a string and return that as the
response of the method. It was a simple method to dynamically change some
xml in a response based on a specific domain name of the requester ....
which is determined by the initial external request on another external
method below :
def GrabOned(self,REQUEST,RESPONSE):
"""Grab a different oned based on url in name"""
RESPONSE.setHeader('content-type','text/xml')
RESPONSE.setHeader('Expires','Fri, 15 Oct 2010 09:00:00 GMT') # matt
being cache friendly .. query strings change via the client
if REQUEST['HTTP_HOST'] ==
"volvo2001snap.sailing.live.virtualspectator.com" :
txt = getattr(self,'oned_volvoHQ')(REQUEST,RESPONSE)
else :
txt = getattr(self,'oned_std').content
return txt
These are all very hacky in nature ... but they serve the purpose for now
while we refactor our thinking.
A call may look like /live/oned.xml?big-client-generated-query-string
where oned.xml is bound to the External Method GrabOned
Now in answer to you other questions :
> It sounds like you've already got a line on where the problem is.
Perhaps I see if I can get it to do the same thing running in the foreground
with -D and put an import pdb and pdb.set_trace() and then sit and wait to
see if anything can get through, even when not busy. Or am I definately
forcing Zope into single thread mode in that instance? I use pdb all the
time when debuggin my products and methods and I think I remember seeing
other requests being honoured. Anyway, not sure how non-linear that would
end up being.
> What does the external method do?
As above
How long does it take if there is
> no load whatsoever?
about 1-2 seconds at the most when not loaded, and now that I look at
things, up to 20 seconds when very loaded, hmm, that could be a clue, I
should get more emperical evidence for that.
How often does it get called?
When those particular people are using their client application, about every
40 seconds.
Can you speed it
> up?
>
Yes I can speed it up, and I can also change the scripts on a dev server so
that I can pretend to be these clients. BTW : I have a completely identical
setup on a dev server, so I really can hack into things without ruining the
live one, if you have things you want me to try that is.
regards
Matt
----- Original Message -----
From: "Chris McDonough" <chrism@zope.com>
To: "Matt" <matt@virtualspectator.com>; <zodb-dev@zope.org>
Sent: Wednesday, November 28, 2001 3:25 PM
Subject: Re: [ZODB-Dev] Zope or Zeo hanging up sockets
> It sounds like you've got a repeatable situation there. I'd be very
> interested to try to help you get to the bottom of this. We've got a
> customer where this symptom tends to happen as well (although less
> predictably).
>
> It sounds like you've already got a line on where the problem is.
> What does the external method do? How long does it take if there is
> no load whatsoever? How often does it get called? Can you speed it
> up?
>
> ----- Original Message -----
> From: "Matt" <matt@virtualspectator.com>
> To: <zodb-dev@zope.org>
> Sent: Tuesday, November 27, 2001 9:06 PM
> Subject: [ZODB-Dev] Zope or Zeo hanging up sockets
>
>
> > I am getting a really strange behavior with requests with Zope on
> ZEO. Not
> > sure what part of the equation is to blame. Our site's busy time
> consists
> > of about 20-40 requests per second, mainly because we have some
> heavy caches
> > in front of us. Sometimes we are finding that the Zope process
> suddenly
> > accelerates to 99.9% CPU and stays there. A sockstat shows that
> connections
> > are piling up to Zope. I watched the logs for a while and found
> that when a
> > certain external method was called that takes up to 10 seconds to
> run and
> > return, that all the other requests pile up behind it. When it
> finally goes
> > through then a burst of some, but not all, of other pending requests
> goes
> > through, I am guessing as many as can fit before the method has been
> called
> > again. To me this seems as if there is only one thread of python
> processing
> > these requests, though in other instances I know for sure that a
> longer
> > process, on the order of minutes, is being processed by Zope, and I
> can
> > still see requests coming through. I was wondering where to turn to
> for
> > debugging. The following is my setup :
> >
> > At present I am running ZEO on a UNIX socket as I only have one Zope
> looking
> > at it right now, but not for long. I have also switched back to
> Data.fs
> > storage to get the speed increase, but was experiencing the same
> socket
> > lockup while using bdb storage.
> >
> > Zope Version (Zope 2.4.0 (source release, python 2.1, linux2),
> python 2.1.0,
> > freebsd4)
> > Python Version 2.1 (#1, Jul 18 2001, 17:06:28) [GCC 2.95.3 [FreeBSD]
> > 20010315 (release)]
> > System Platform freebsd4
> >
> > the guts of me start scripts are as follows :
> >
> > Zope :
> >
> > exec chroot /usr/local/sandbox /usr/local/bin/python \
> > $INST_HOME/z2.py \
> > -m '' -f "${FTPPORT}" -a ${MYIP} -p '' -w "${WWWPORT}" -F '' \
> > -Z ''
> >
> > Zeo :
> >
> > exec chroot /usr/local/sandbox /usr/local/bin/python \
> > $INST_HOME/lib/python/ZEO/start.py -U $INST_HOME/var/zeo.soc \
> > -S main=StorageConfig:data_storage \
> > -S bdb=StorageConfig:bdb_storage \
> > ZEO_SERVER_PID=$INST_HOME/var/ZEO_SERVER.pid \
> > -s "$@"
> >
> > regards
> > Matt
> >
> >
> >
> >
> > _______________________________________________
> > For more information about ZODB, see the ZODB Wiki:
> > http://www.zope.org/Wikis/ZODB/
> >
> > ZODB-Dev mailing list - ZODB-Dev@zope.org
> > http://lists.zope.org/mailman/listinfo/zodb-dev
> >