[ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)

[Chris Withers]

Okay, now that we have 2.8.4 in place, we get proper reporting of 
ConflictErrors and today we started seeing one happening over and over 
again which looked roughly as follows:

Traceback (most recent call last):
   File "lib/python/Products/Transience/Transience.py", line 844, in 
new_or_existing
     item = self.get(key, _marker)
   File "lib/python/Products/Transience/Transience.py", line 491, in get
     item = self._move_item(k, current_ts, default)
   File "lib/python/Products/Transience/Transience.py", line 346, in 
_move_item
     if getattr(self._data[current_ts][k], 'setLastAccessed', None):
   File "lib/python/ZODB/Connection.py", line 704, in setstate
     self._setstate(obj)
   File "lib/python/ZODB/Connection.py", line 760, in _setstate
     self._reader.setGhostState(obj, p)
   File "lib/python/ZODB/serialize.py", line 495, in setGhostState
     state = self.getState(pickle)
   File "lib/python/ZODB/serialize.py", line 488, in getState
     return unpickler.load()
   File "lib/python/ZODB/serialize.py", line 436, in _persistent_load
     return self._conn.get(oid)
   File "lib/python/ZODB/Connection.py", line 207, in get
     p, serial = self._storage.load(oid, self._version)
   File "lib/python/ZEO/ClientStorage.py", line 746, in load
     return self.loadEx(oid, version)[:2]
   File "lib/python/ZEO/ClientStorage.py", line 769, in loadEx
     data, tid, ver = self._server.loadEx(oid, version)
   File "lib/python/ZEO/ServerStub.py", line 192, in loadEx
     return self.rpc.call("loadEx", oid, version)
   File "lib/python/ZEO/zrpc/connection.py", line 536, in call
     raise inst # error raised by server
ConflictError: database conflict error (oid 0x010f0d)

The things which got me worried is that it was always the same oid that 
was conflicting, and that's not usually something we see. We were also 
getting it very frequently, and on both reads and writes. That in itself 
is enough to get me to mail here. I had a dig on the ZEO server and 
found the matching traceback for one of them there:

------
2006-01-24T12:42:57 INFO ZEO.zrpc.Connection(S) (x.x.x.x:y)
loadEx()  raised exception: database conflict error (oid 0x010f0d)
Traceback (most recent call last):
   File "lib/python/ZEO/zrpc/connection.py",line 421, in handle_request
     ret = meth(*args)
   File "lib/python/ZEO/StorageServer.py",line 248, in loadEx
     return self.storage.loadEx(oid, version)
   File"lib/python/tempstorage/TemporaryStorage.py", line 132, in loadEx
     data = self.load(oid, version)
   File"lib/python/tempstorage/TemporaryStorage.py", line 117, in load
     raise POSException.ConflictError(oid=oid)
ConflictError: database conflict error (oid 0x010f0d)

Now, I wanted to find out what the troublesome object behind 0x010f0d 
was, so I fired up and debug zope client, and was distrubed by the result:

 >>> app.temp_folder._p_jar[p64(0x010f0d)]
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "lib/python/ZODB/Connection.py", line 207, in get
     p, serial = self._storage.load(oid, self._version)
   File "lib/python/ZEO/ClientStorage.py", line 746, in load
     return self.loadEx(oid, version)[:2]
   File "lib/python/ZEO/ClientStorage.py", line 769, in loadEx
     data, tid, ver = self._server.loadEx(oid, version)
   File "lib/python/ZEO/ServerStub.py", line 192, in loadEx
     return self.rpc.call("loadEx", oid, version)
   File "lib/python/ZEO/zrpc/connection.py", line 536, in call
     raise inst # error raised by server
ZODB.POSException.ConflictError: database conflict error (oid 0x010f0d)

We managed to stop the problem causing furthe conflict errors by 
fiddling with the timeout resolution on the Transient Object Container 
at /temp_folder/session_data through the web. I guess this re-created 
the data structures used by the temp_folder or session_data objects and 
so took the object with the dodgy oid out of the equation.

However, even now, an hour or so later, if I fire up a debug client and 
try and look up that oid, I get a ConflictError. Any idea what's going on?

cheers,

Chris