[ZODB-Dev] database conflict fail

[Claudiu Saftoiu]

Hello ZODB List,

(This is also a stackoverflow question - you might prefer the formatting
there:
http://stackoverflow.com/questions/9810116/zodb-database-conflict-fail )

I have a server, and a client.

A client sends a request. The request has a certain key associated with it,
e.g. `a-1`, `a-2`, `b-1`, `b-4`.

If two requests for the same key come in at once, there will be a conflict
error, as the same data structures are being modified.

I can adapt the client to simply not send two requests of the same key at
once. However, I'd like this system to work with multiple clients, too. It
seems silly to have the clients coordinate what they send to the server.
Instead, I'd like the server to simply block on a request of a certain key
if that key is already being modified, until the other requests with that
same key are done.

To this end, I've created a locking system. At the beginning of the
function on the server, I do:

    key = ...
    print "Acquiring %s lock..." % (key,)
    KEY_LOCKS[key].acquire()
    print "%s lock acquired." % (key,)
    def after_commit_hook(success):
        KEY_LOCKS[key].release()
        print "(after %s commit): Released %s lock" % (('failed',
'successful')[success], key)
    transaction.get().addAfterCommitHook(after_commit_hook)

where `KEY_LOCKS` is a dict mapping keys to `threading.Lock`s. Afterwards
follows the code that modifies the persistent data structures.

What I assume would happen is that, if a request comes in for a key that's
already being processed, it would block when acquiring the lock. Only when
the earlier request **has already been committed** (thus being beyond any
conflict errors), would the new request resume. The requests do nothing
that would conflict until the lock is acquired.

Most of the requests work fine:

    Acquiring a-b lock...
    a-b lock acquired.
    (after successful commit): Released a-b lock
    Acquiring a-c lock...
    a-c lock acquired.
    (after successful commit): Released a-c lock

However, there is _still_ an issue when the same key is sent, even though
the locking seems to work:

    Acquiring q-q lock...
    q-q lock acquired.
    Acquiring q-q lock...
    (after successful commit): Released q-q lock
    q-q lock acquired.
    (after failed commit): Released q-q lock
    repoze.retry retrying, count = 1
    Traceback (most recent call last):
    ...
    ConflictError: database conflict error (oid 0x13009b, class
persistent.list.PersistentList)

And then the request retries. Note that the `q-q lock` was only acquired
after the successful commit.

What gives? Why is this system not preventing conflict errors? Where is my
assumption incorrect?

---

If, before the `transaction.get().addAfterCommitHook(after_commit_hook)`
line I put `transaction.begin()`, it works. For the life of me I can't
figure out why. Before the `transaction.begin()` line, the entirety of my
code is:

    post = request.params
    if not post: return Response("No data!")

    data = eval(post['data'])
    time_parsed = time.time()
    my_app = request.context

This solves my problem but I'm not putting that as an answer 'cause I still
want to know: Why does it give conflict errors if I don't start a fresh
transaction right before?

Thanks all,
- Claudiu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20120321/1b351632/attachment.html>