[Zope] ZEO disconnects, Zope auto restarts (via zopectl)
allison at shasta.stanford.edu
Fri Feb 3 11:35:11 EST 2006
Thanks for the assist. Comments on your comments interlinearly below.
I have increased cache and other resources to see what the impact will be.
On Fri, 3 Feb 2006, Paul Winkler wrote:
> On Fri, Feb 03, 2006 at 01:00:45AM -0800, Dennis Allison wrote:
> > Zope 2.9.0
> > We are seeing spontaneous restarts of Zope with no indication in any of
> > the standard Zope logs. Looking at the ZEO log indicates that the
> > restarts of Zope are due to a lost connection between Zope & ZEO but
> > with no other information. The logging level is set at the distribution
> > default (INFO).
> Are you *sure* that is the cause, rather than the effect?
No, I am not and there's nothing in the logs which hints at why
it restarted. We are running under load. The failures are
silent. We do have a fairly high rate of conflict errors (which
all get resolved finally!).
> If zope restarts for any reason, I'd expect the zeo log to
> show a disconnect and reconnect as a result.
> Check the clocks on your zope and zeo boxes and make sure the timing of
> events in your logs is really what you think it is. (Systems that
> aren't running ntpd are the bane of my existence...)
Timing correlates to the second. Zope and ZEO live on the
same physical box.
> Wild guess: Any chance your Zope process is running out of
> memory? I've had that on several occasions, when some
> naively-written software attempts to do something huge in
> memory that should really use a temp file on disk.
> (Zope itself used to have some code like that in the FTP
> server, don't know if it still does.)
I doubt if I am hitting a limit. The box has nearly 8GB of
memory most of which (6GB) is used by linux as a cache.
No messages in the logs.
> I discovered this by looking in /var/log/messages.
> At least on linux, the kernel will log something there
> when it kills a process that consumes all available memory.
> > We are running a fairly vanilla setup, excerpted below:
> > [snip...]
> Unrelated to your problem, and maybe you know this, but depending on the
> size of your storage, I'd consider increasing the zeo client cache size.
> It's a disk cache and you can safely make it huge. But if you don't see
> "cache flipping" messages in your event log, it may not matter.
Done, but I cannot report on the effect.
More information about the Zope