[Zope] ZEO disconnects, Zope auto restarts (via zopectl)

[Paul Winkler]

On Fri, Feb 03, 2006 at 01:00:45AM -0800, Dennis Allison wrote:
> 
> Zope 2.9.0
> 
> We are seeing spontaneous restarts of Zope with no indication in any of 
> the standard Zope logs.  Looking at the ZEO log indicates that the 
> restarts of Zope are due to a lost connection between Zope & ZEO but 
> with no other information.  The logging level is set at the distribution 
> default (INFO).

Are you *sure* that is the cause, rather than the effect?  

If zope restarts for any reason, I'd expect the zeo log to 
show a disconnect and reconnect as a result.

Check the clocks on your zope and zeo boxes and make sure the timing of
events in your logs is really what you think it is.  (Systems that
aren't running ntpd are the bane of my existence...)

Wild guess: Any chance your Zope process is running out of
memory?  I've had that on several occasions, when some
naively-written software attempts to do something huge in
memory that should really use a temp file on disk.
(Zope itself used to have some code like that in the FTP
server, don't know if it still does.)

I discovered this by looking  in /var/log/messages.
At least on linux, the kernel will log something there
when it kills a process that consumes all available memory.

> We are running a fairly vanilla setup, excerpted below:
> 
> <<zope.conf>>
> # ZEO client storage:
> #
> <zodb_db main>
>   mount-point /
>   # ZODB cache, in number of objects
>   cache-size 5000
>   <zeoclient>
>     server localhost:8301
>     storage 1
>     var $INSTANCE/var
>     # ZEO client cache, in bytes
>     cache-size 20MB

Unrelated to your problem, and maybe you know this, but depending on the
size of your storage, I'd consider increasing the zeo client cache size.
It's a disk cache and you can safely make it huge.  But if you don't see
"cache flipping" messages in your event log, it may not matter.

>     # Uncomment to have a persistent disk cache
>     client group1-zeo
>   </zeoclient>
> </zodb_db>
> 
> <<zeo.conf>>
> <zeo>
>   address localhost:8301
>   read-only false
>   invalidation-queue-size 100
>   pid-filename $INSTANCE/var/ZEO.pid
>   # monitor-address PORT
>   # transaction-timeout SECONDS
> </zeo>
> 
> <runner>
>   program $INSTANCE/bin/runzeo
>   socket-name $INSTANCE/etc/zeo.zdsock
>   daemon true
>   forever false
>   backoff-limit 10
>   exit-codes 0, 2
>   directory $INSTANCE
>   default-to-interactive true
>   # user zope
>   python /usr/bin/python2.4
>   zdrun /usr/local/src/zope/Zope2.9/lib64/python/zdaemon/zdrun.py
> 
>   # This logfile should match the one in the zeo.conf file.
>   # It is used by zdctl's logtail command, zdrun/zdctl doesn't write it.
>   logfile $INSTANCE/log/zeo.log
> </runner>
> 
> It's not clear what changes will lead to a more stable connection because 
> it is not clear what's triggering the problem.  Any advice would be 
> appreciated.
> 
> Presumably the shotgun approach would work -- increase the cache sizes, 
> lengthen the invalidation-queue-size, and increase the backoff-limit but 
> it would be nice to have some guidance.
> 
> 5~
> 
> 
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )

-- 

Paul Winkler
http://www.slinkp.com