[ZODB-Dev] weird conflicts with BTrees

[Stephan Richter]

On Wednesday, November 30, 2011 11:44:20 AM Adam GROSZER wrote:
> Hello,
> 
> I thought BTrees have conflict resolution code.
> 
> We have here a weird conflict here, the BTree in question is a
> SessionDataContainer. The conflict seems to appear when one transaction
> removes and the other adds an entry.

I am working with Adam on this, so here is some more detail. First of all, 
this clearly does not happen all the time. It seems to happen only 
infrequently, which I think is in cases where the buckets of the BTree get 
reorganized due to a deletion sweep.

I think the reason why this is a relatively common scenario is:

1. There is a lot of activity on the site at some point.
2. Then there is a prolonged period of inactivity, which causes a bunch of 
sessions to time out.
3. Then activity is starting again, causing the first transaction to do a large 
session removal sweep (which causes the BTree restructuring) and the next 
concurrent transaction to add a new session causing a conflict.

At least for sessions, conflict resolution is not that difficult, because we are 
always dealing with unique keys and keys are never re-assigned to another 
object. So it effectively becomes a set conflict resolution issue.

Also, I think that BTrees might be overkill for this problem, since the amount 
of active authenticated users on a system is usually small. And even if there 
is a large amount of users, the key->session mapping will always reside in the 
ZODB object cache (though there is some overhead to distributing new versions 
of the mapping due to the relatively high number of writes.)

I am tempted to write a conflict resolving dictionary for this, but I would 
love hear whether someone has done something like this already.

Regards,
Stephan
-- 
Entrepreneur and Software Geek
Google me. "Zope Stephan Richter"