[ZODB-Dev] ZEO Cache inconsistency

[Vincent Pelletier]

Hi.

I'm facing a cache consistency problem in an old Zeo running on 2.8.8 
products.

Minimal required setup:
- Zeo sharing a ZODB
- MySQL
- Zope instance 1 mounting the shared ZODB and connected to MySQL (Z1)
- Zope instance 2 mounting the shared ZODB and connected to MySQL (Z2)

Scenario:
- Z1 & Z22 access some object to be used as container in next step, so both
  have it in cache
- Z1 creates an object in ZODB, and stored its path in MySQL
- Z2 fetches the path from MySQL and tries to fetch the object from the
  ZODB and gets a KeyError

Traced cause:
Z2 didn't process the cache invalidation request sent to it by Zeo upon 
transaction beginning (well, upon joining the database connector).
If I access the new subobject by its OID from Z2 during the 3rd scenario step, 
it works (object is found in Zeo).
If I peek at pending invalidations from Z2 during the 3rd scenario step, I 
found an invalidation for the container.

Note: in my test case, the container is a BTreeFolder2, but this should in 
theory happen with any kind of container. BTreeFolder2 makes the "invalidation 
peek" check harder, as one must find the OID of the bucket which contains the 
new object.

Note2: this bug of course does not happen each time. It was triggered with an 
approx. 1 per 5000 ratio.

Could you tell me if there is a fix for this problem in more recent versions 
of Zeo ?

Do you have any idea of a possible fix ?

I believe
- this can only be avoided by adding an exchange with Zeo servers so Zope can
  check if it processed all invalidations prior to joining the storage, so
  storage would "force" Zope to process pending invalidations
- it is not acceptable to do such check upon object read access (too heavy)
- it might be hard to do that exchange at the end of the transaction (the last
  invalidation number will have been long forgotten by Zeo)
- it might slow down transaction to add an exchange at storage join (if there
  is none at the moment, which I don't know)

-- 
Vincent Pelletier