[ZODB-Dev] Ordering before commit hooks
Jim Fulton
jim at zope.com
Tue Aug 30 15:30:50 EDT 2005
Julien Anguenot wrote:
I'm gonna respond to a number of emails at once. :)
> Tim Peters wrote:
>
>>[Tim]
>>
>>
>>>...
>>>Julien provided links to code that already uses the new feature:
>>>
>>>""" As an Indexation Manager :
>>>http://svn.nuxeo.org/trac/pub/file/CPSCore/trunk/IndexationManager.py
>>>
>>>As an Event Manager :
>>>http://svn.nuxeo.org/trac/pub/file/CPSSubscriptions/trunk/EventManager.py
>>>"""
>>>
>>>He appears to use (just) two distinct `order` levels there, and seems
>>>just to want to make sure one class of hook gets run before the other
>>>class of hook. The new scheme does give an easy way to do that.
>>
>>
>>OTOH, while picking levels of -100 and 100 works for that specific use case,
>>looks like it threatens to become a mess if multiple subsystems try to use
>>this scheme simultaneously.
>
>
> We discussed quickly about this with Florent. This can be expressed by
> having a coding guide for the given framework (for instance CPS). We
> will provide ranges in which hooks are executed. Then if someone wants
> to register another one he can check the existing ones and the ranges
> they are using and what they are doing. Just a matter of documentation.
Of course, this doesn't work across projects.
>
>>For example, someone who wants to be "the last
>>kind of hook invoked" has no way to force that, at least not short of
>>passing sys.maxint as the `order`. But if that's what's needed,
>>`sys.maxint` is a strange way to spell it.
>
>
> I understand it but this is something that could be done by extending
> the transaction hooks sub-system. But again we don't have YAGNI yet for
> a bigger sub-system that could include this.
The very fact that to really do what you need to do today would require an
extension indicates that the current API isn't what we should be doing.
We'll never have what every application needs anyway, which is why this
ordering logic should be at the application level where it's easy to control.
> I don't understand why keeping the order paremeter is such a big deal ?
Because it clutters the ZODB API with something that is, fundamentally
application policy.
> It prevents me having to code my own specific object dealing with this
Dealing with what? Your original need was to make something go last.
As Tim pointed out, the order api doesn't do that and, as I've pointed out,
the existing API does deal with that.
> and I'm sure other people will benefit from this later on when they will
> have to use this transaction feature.
That's hypothetical. In any case, if you are so sure, provide a system
that does this and let people use it. You don't need to make ZODB more
complicated to address this use case.
Julien Anguenot wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jim Fulton wrote:
...
>>Minor note, AFAICT, Julien really only needed to assure that his event hook
>>ran last. He could have done this easily without the order feature by
>>registering
>>an intermediate hook that registered the event hook when it was called.
>
>
> Sure Jim. I could have done this but this is something that, I find,
> would be overkill for such a simple use case.
This solution is simple. Instead, you modified the ZODB APIs in a way that
doesn't actually address the use case. This isn't even overkill. It's
just kill. ;)
>>Another way to say this is that it pushes application policy into ZODB.
>>Different applications will likely need other policies.
>
>
> (see my previous post to Tim) I called it documentation but I ment
> policy which removes the hook registration order ranges problems Tim
> mentionned.
No, the policy of using an order is still pushed down to ZODB. You want
to order callbacks using an integer. Someone else might want to
order callbacks based on some sort of type. Someone else might
want a scheme for removing "duplicate callbacks". These are all reasonable
use cases. Just because they are reasonable use cases, doesn't mean they
have to be handled by ZODB. They can and shood be handled by higher-level
components.
Dieter Maurer wrote:
> Tim Peters wrote at 2005-8-22 16:48 -0400:
>
>>...
>>Jim still wonders, and he got me wondering too, whether the `order=` gimmick
>>is really needed.
>
>
> But, it is a very easy concept -- both easy to grasp as well as easy
> to implement.
Just because it's an easy concept, doesn't mean it belongs in
in the transaction layer. It would be just as easy a concept
as a ZODB add-on. I wouldn't object to such a thing in Zope, for
example.
>
>>For example, you could have gotten to the same end here
>>with the old method, by registering your actions with an object of your own
>>creation, and registering just one commit hook with the transaction, where
>>that one hook looked at the actions you registered with your own object and
>>ran them in whatever order _it_ determined was best.
>
>
> Now plug and play comes into play:
>
> Assume two packages developped by independent people
> which all want to control the order of hook execution.
>
> Each package can proceed as you propose. But, in case
> there are order dependencies, a global ordering is lacking.
>
> With the "order" gimmick, the global ordering can be
> obtained with a set of parameters, controlled by ZCML.
This is a lot of machinery to use the gimmick. Now you have to
have ZCML directives to support order configuration.
Adding an intermediate hook that supports ordering is trivial
compared to the ZCML configuration machinery.
Julien Anguenot wrote:
> Jim Fulton wrote:
...
>>Do we have evidence that such applications exist? So far, the
>>only example I've seen is one where an application wanted a handler
>>to go last. As I've pointed out in a separate note, this is achievable
>>without the change.
>>
>
>
> We have. We are thinking of adding another hook within CPS to maintain
> our tree caches. (I'm not going into details on the CPS tree cache here)
>
> We would have these 3 hooks that we will want to see executed in the
> following order :
>
> IndexationManager -> TreeCacheManager -> EventManager
>
> If you want more information about this Jim, feel free to ask.
Why does the TreeCacheManager need to run after the indexation
manager? Does it use results from the indexation manager? if so,
why can't it call the indexation manager?
In any case, I don't want to tell you what your application needs.
But you don't need to modify ZODB to get what it needs.
Florent Guillaume wrote:
> Just back from vacation...
I hope it was rejuvinating. :)
>
> Jeremy Hylton wrote:
>
>> On 8/22/05, Tim Peters <tim at zope.com> wrote:
>>
>>> Jim still wonders, and he got me wondering too, whether the `order=`
>>> gimmick
>>> is really needed. For example, you could have gotten to the same end
>>> here
>>> with the old method, by registering your actions with an object of
>>> your own
>>> creation, and registering just one commit hook with the transaction,
>>> where
>>> that one hook looked at the actions you registered with your own
>>> object and
>>> ran them in whatever order _it_ determined was best. The ordering logic
>>> would have been out of ZODB then, not limited to what an integer
>>> `order` can
>>> express, and might even benefit from "ah, if I have to run A, then
>>> there's
>>> no need to also run B or C" kinds of optimizations.
>
>
> But that only works if you have all the code in a single framework. If
> Zope had a commit hook, and that CPS wanted to add another one that "ran
> after Zope's", there would be no way to do it.
Right, they have to agree on a common API. With the order gimmic,
this is a ZODB API. It could as easily be a higher-level API.
Since CPS in built on Zope, it could even be a Zope API that
CPS used.
>> I think that's the right reasoning. I agree with Jim.
>>
>> The transaction manager coordinates the actions of unconnected
>> resource managers. If there are several transaction participants that
>> are all part of the same software package, they can provide their own
>> internal ordering as you suggest. If they are not related, then
>> there's no reason to think they care about their order relative to
>> other participants they know nothing about. To the extent that
>> software cares about order, there is likely a simple partial order
>> (run before X) rather than the total order that order= suggests to me.
>
>
> Yes I agree but the partial order makes the implementation much more
> complex than a global agreed-on integer scale (you have to deal with the
> naming of packages, potential loops, etc). I agree though that there has
> to be some "rough consensus" on what integer mean what (i.e., well-known
> values for important frameworks).
Right, I don't see integers as a good way of ordering things. I also
really believe that most applications will be as well served by a facility
that allows things to go last -- and we can already do that,
Sadly, the inspiration for the order gimmic was the order hack
(yes hack ;) I put in zope.configure because I wanted to provide a way
to delay some actions until after all non-delayed actions. This was
another case where last was what I wanted. (I used much better orders,
9999999, BTW. ;)
Julien Anguenot wrote:
...
> I understand this argument Jeremy. But check Dieter argument about the
> ability of registering hooks with zcml directive for instance.
That's a way to coordinate order among applications. It's also
rather heavy. I'm very unconvinced that separate applications
need to coordinate order. If they do, then I suspect we will find
better ways to do it once we have an actual need.
> The good think by keeping the order parameter is that you can use *both*.
You can't do both within ZODB because ZODB only supports integer order.
If it's acceptable to use a dependency/partial order approach as an
add on, then why isn't it *also* acceptable to do the ordering
approach as an add on?
> The 'order' parameter provided by the transaction *and* the ability of
> coding your own object managing this if your system has a complex use of
> the before commit hooks. (at least more complex than *just* the order of
> the hooks) You may think about execution dependencies for instance.
> Then it would the responsability of the framework (on top of ZODB) to
> manage its complex logic. This use case will be rare and really complex
> framework will deal with that in their internals. Here, I agree with you.
I think that's a good approach. Let's use it for ordering too.
> But when an integrator (or devel) will have to register a commit hook he
> will be happy to be able to control easily the order of its hooks
> without having to think about coding it's object managing this.
Huh? Why would a developer do this in isolation from other
frameworks. If he cares about order other than going last, then
he's got much bigger problems than actually implementing an ordering
scheme. He's got to coordinate with other developers so that his
hook gets called in the right order.
> Again,
> it's overkill in simple situations.
The only situation we have is one where something wants to go
last and the order gimmic is overkill for that (at best, or just
inadequate).
> You're not convincing me for simple situations
I'm increasingly convinced that hook ordering should be provided by an
add-on framework.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
More information about the ZODB-Dev
mailing list