[ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)

[Chris McDonough]

Rather than chucking this in the collector as-is, it would be useful  
if you could get me the output of the ZEO log rather than the output  
of the client log.  Right now, the actual traceback is in the ZEO  
log; the client log only shows that a conflict error happened; not  
where it happened.

FWIW, I've not tested ZEO+tempstorage, as in my own deployments I  
always use "session affinity" which always sends subsequent requests  
to the same appserver that received a request containing a particular  
session cookie.

Thanks!

- C


On Jan 24, 2006, at 8:41 AM, Chris Withers wrote:

> Okay, now that we have 2.8.4 in place, we get proper reporting of  
> ConflictErrors and today we started seeing one happening over and  
> over again which looked roughly as follows:
>
> Traceback (most recent call last):
>   File "lib/python/Products/Transience/Transience.py", line 844, in  
> new_or_existing
>     item = self.get(key, _marker)
>   File "lib/python/Products/Transience/Transience.py", line 491, in  
> get
>     item = self._move_item(k, current_ts, default)
>   File "lib/python/Products/Transience/Transience.py", line 346, in  
> _move_item
>     if getattr(self._data[current_ts][k], 'setLastAccessed', None):
>   File "lib/python/ZODB/Connection.py", line 704, in setstate
>     self._setstate(obj)
>   File "lib/python/ZODB/Connection.py", line 760, in _setstate
>     self._reader.setGhostState(obj, p)
>   File "lib/python/ZODB/serialize.py", line 495, in setGhostState
>     state = self.getState(pickle)
>   File "lib/python/ZODB/serialize.py", line 488, in getState
>     return unpickler.load()
>   File "lib/python/ZODB/serialize.py", line 436, in _persistent_load
>     return self._conn.get(oid)
>   File "lib/python/ZODB/Connection.py", line 207, in get
>     p, serial = self._storage.load(oid, self._version)
>   File "lib/python/ZEO/ClientStorage.py", line 746, in load
>     return self.loadEx(oid, version)[:2]
>   File "lib/python/ZEO/ClientStorage.py", line 769, in loadEx
>     data, tid, ver = self._server.loadEx(oid, version)
>   File "lib/python/ZEO/ServerStub.py", line 192, in loadEx
>     return self.rpc.call("loadEx", oid, version)
>   File "lib/python/ZEO/zrpc/connection.py", line 536, in call
>     raise inst # error raised by server
> ConflictError: database conflict error (oid 0x010f0d)
>
> The things which got me worried is that it was always the same oid  
> that was conflicting, and that's not usually something we see. We  
> were also getting it very frequently, and on both reads and writes.  
> That in itself is enough to get me to mail here. I had a dig on the  
> ZEO server and found the matching traceback for one of them there:
>
> ------
> 2006-01-24T12:42:57 INFO ZEO.zrpc.Connection(S) (x.x.x.x:y)
> loadEx()  raised exception: database conflict error (oid 0x010f0d)
> Traceback (most recent call last):
>   File "lib/python/ZEO/zrpc/connection.py",line 421, in handle_request
>     ret = meth(*args)
>   File "lib/python/ZEO/StorageServer.py",line 248, in loadEx
>     return self.storage.loadEx(oid, version)
>   File"lib/python/tempstorage/TemporaryStorage.py", line 132, in  
> loadEx
>     data = self.load(oid, version)
>   File"lib/python/tempstorage/TemporaryStorage.py", line 117, in load
>     raise POSException.ConflictError(oid=oid)
> ConflictError: database conflict error (oid 0x010f0d)
>
> Now, I wanted to find out what the troublesome object behind  
> 0x010f0d was, so I fired up and debug zope client, and was  
> distrubed by the result:
>
> >>> app.temp_folder._p_jar[p64(0x010f0d)]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "lib/python/ZODB/Connection.py", line 207, in get
>     p, serial = self._storage.load(oid, self._version)
>   File "lib/python/ZEO/ClientStorage.py", line 746, in load
>     return self.loadEx(oid, version)[:2]
>   File "lib/python/ZEO/ClientStorage.py", line 769, in loadEx
>     data, tid, ver = self._server.loadEx(oid, version)
>   File "lib/python/ZEO/ServerStub.py", line 192, in loadEx
>     return self.rpc.call("loadEx", oid, version)
>   File "lib/python/ZEO/zrpc/connection.py", line 536, in call
>     raise inst # error raised by server
> ZODB.POSException.ConflictError: database conflict error (oid  
> 0x010f0d)
>
> We managed to stop the problem causing furthe conflict errors by  
> fiddling with the timeout resolution on the Transient Object  
> Container at /temp_folder/session_data through the web. I guess  
> this re-created the data structures used by the temp_folder or  
> session_data objects and so took the object with the dodgy oid out  
> of the equation.
>
> However, even now, an hour or so later, if I fire up a debug client  
> and try and look up that oid, I get a ConflictError. Any idea  
> what's going on?
>
> cheers,
>
> Chris
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev
>