[ZODB-Dev] possible deadlock with Relstorage + Oracle ?

[Mauro Amico]

Hi there,

We currently use Plone 4.2 with Relstorage and Oracle RAC, sometimes 
(like once a week) it
happens that Zope threads remains locked and Zope stops to serve new 
requests. When that happens, it is frequent that the phenomenon appears 
over all the instances, on different servers.

Currently we have configured 2 Zope threads per instance. In attach 
there is log reported by haufe.requestmonitoring,
with traces, always the same, when the instances stop to work.

As first action I'm thinking to increase to three threads per instance, 
other ideas or directions to investigate?

Regards,
Mauro.

Products:
------------

* RelStorage-1.5.1-py2.7
* cx_Oracle-5.1.2-py2.7-linux-x86_64
* Oracle Instance Client Shared Library 64-bit - 11.2.0.3.0
* ZODB3 3.10.5-py2.7-linux-x86_64
* Zope2 2.13.19-py2.7

Logs:
-------

2013-11-25T13:10:52 WARNING RequestMonitor.DumpTrace Long running request
Request 583318 "/@@healthcheck" running in thread 139854656571136 since 
28.9825799465s
Python call stack (innermost first)
   Module relstorage.adapters.oracle, line 322, in restart_load
   Module relstorage.storage, line 252, in _restart_load_and_call
   Module relstorage.storage, line 1204, in _restart_load_and_poll
   Module relstorage.storage, line 1230, in poll_invalidations
   Module ZODB.Connection, line 492, in _flush_invalidations
   Module ZODB.Connection, line 1045, in open
   Module ZODB.DB, line 751, in open
   Module App.ZApplication, line 43, in __bobo_traverse__
   Module ZPublisher.BaseRequest, line 406, in traverse
   Module ZPublisher.Publish, line 49, in publish
   Module ZPublisher.Publish, line 7, in _patched_publish
   Module ZPublisher.Publish, line 237, in publish_module_standard
   Module ZPublisher.Publish, line 443, in publish_module
   Module ZServer.PubCore.ZServerPublisher, line 31, in __init__

2013-11-25T13:10:52 WARNING RequestMonitor.DumpTrace Long running request
Request 583319 "/@@healthcheck" running in thread 139854648178432 since 
28.5510978699s
Python call stack (innermost first)
   Module threading, line 127, in acquire
   Module ZODB.DB, line 725, in open
   Module App.ZApplication, line 43, in __bobo_traverse__
   Module ZPublisher.BaseRequest, line 406, in traverse
   Module ZPublisher.Publish, line 49, in publish
   Module ZPublisher.Publish, line 7, in _patched_publish
   Module ZPublisher.Publish, line 237, in publish_module_standard
   Module ZPublisher.Publish, line 443, in publish_module
   Module ZServer.PubCore.ZServerPublisher, line 31, in __init__