[Checkins] SVN: Products.QueueCatalog/trunk/ moved
Andreas Jung
andreas at andreas-jung.com
Tue May 13 06:37:51 EDT 2008
Log message for revision 86691:
moved
Changed:
D Products.QueueCatalog/trunk/CHANGES.txt
D Products.QueueCatalog/trunk/CatalogEventQueue.py
D Products.QueueCatalog/trunk/CatalogEventQueueSet.py
A Products.QueueCatalog/trunk/Products/QueueCatalog/CHANGES.txt
A Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueue.py
A Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueueSet.py
A Products.QueueCatalog/trunk/Products/QueueCatalog/dtml/
A Products.QueueCatalog/trunk/Products/QueueCatalog/help/
A Products.QueueCatalog/trunk/Products/QueueCatalog/version.txt
A Products.QueueCatalog/trunk/Products/QueueCatalog/www/
D Products.QueueCatalog/trunk/dtml/
D Products.QueueCatalog/trunk/help/
D Products.QueueCatalog/trunk/version.txt
D Products.QueueCatalog/trunk/www/
-=-
Deleted: Products.QueueCatalog/trunk/CHANGES.txt
===================================================================
--- Products.QueueCatalog/trunk/CHANGES.txt 2008-05-13 10:36:31 UTC (rev 86690)
+++ Products.QueueCatalog/trunk/CHANGES.txt 2008-05-13 10:37:50 UTC (rev 86691)
@@ -1,81 +0,0 @@
-QueueCatalog Product Changelog
-
- QueueCatalog 1.5 (2007/12/15)
-
- - Replaced validateValue call with proper validate call. This
- makes QueueCatalog compatible with Zope 2.9
-
- QueueCatalog 1.4 (2006/05/09)
-
- - Added a second catalog queue conflict resolution strategy and made
- the strategy configurable via the 'Configure' ZMI tab. The new policy
- can be tried when ZODB conflict errors emanating from the catalog
- queue buckets become a problem. It attempts to avoid conflicts by
- doing even more guessing about what the behavior should be, rather
- than deeming a conflicting situation as too insane to handle and
- raising a ConflictError.
-
- - Removed all uses of zLOG in favor of the Python logging module
-
- - Updated some help screens
-
- - Added a dependencies document
-
- - Updated the tests to run on Zope 2.8+
-
- - Factored out queue conflict testing after changing it to use a quick
- and dirty FileStorage, and expanding it a little.
-
- - Removed the security declaration for a method named refreshCatalog
- which was added in version 1.0 - this is the wrong place to apply
- product-specific settings for a product the QueueCatalog is not
- dependent on.
-
- - CVS-only, tag 'QueueCatalog-post-1_3-error-logging'
-
- - When performing deferred indexing, don't attempt (again) to
- assign brains metadata
-
- - When indexing an object raises an Exception, log it, and remove
- the object from the queue
-
- QueueCatalog 1.3 (2004/11/22)
-
- - Released by ctheune at:
- http://zope.org/Members/ctheune/QueueCatalog/1.3/QueueCatalog-1.3.tar.gz
-
- - CVS tag 'QueueCatalog-1_3'
-
- - QueueCatalog.py: Made the immediate processing of DELETE-Requests
- more efficient. Christian Theune <ct at gocept.com>
-
- - QueueCatalog.py: Proxied ZCatalog.Indexes attribute.
-
- - QueueCatalog.py: Corrected a permission declaration typo for
- 'manage_size'.
-
- - QueueCatalog.py: Modified 'manage_queue' to display a list of queued
- items (limit 100 items).
-
- QueueCatalog 1.2 (2003/11/06)
-
- - CVS-only release, tag 'QueueCatalog-1_2'
-
- - Add uidForObject hook which allows the underlying catalog to specify
- uids for new objects being indexed. If this hook is not implemented,
- uids are abs paths as before.
-
- QueueCatalog 1.1 (2003/08/06)
-
- - CVS-only release, tag 'QueueCatalog-1_1'
-
- - Fix brown-paper bag error which prevented queue processing from
- *ever* working.
-
- QueueCatalog 1.0 (2003/06/25)
-
- - CVS-only release, tag 'QueueCatalog-1_0'
-
- - QueueCatalog.py (QueueCatalog.refreshCatalog): Added
- refreshCatalog method that queues the reindexing. Plone uses this
- method on portal creation. Sidnei da Silva <sidnei at x3ng.com>
Deleted: Products.QueueCatalog/trunk/CatalogEventQueue.py
===================================================================
--- Products.QueueCatalog/trunk/CatalogEventQueue.py 2008-05-13 10:36:31 UTC (rev 86690)
+++ Products.QueueCatalog/trunk/CatalogEventQueue.py 2008-05-13 10:37:50 UTC (rev 86691)
@@ -1,289 +0,0 @@
-##############################################################################
-#
-# Copyright (c) 2002-2006 Zope Corporation and Contributors.
-# All Rights Reserved.
-#
-# This software is subject to the provisions of the Zope Public License,
-# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
-# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
-# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
-# FOR A PARTICULAR PURPOSE.
-#
-##############################################################################
-"""
-$Id$
-"""
-
-import logging
-
-from Persistence import Persistent
-from ZODB.POSException import ConflictError
-
-logger = logging.getLogger('event.QueueCatalog')
-
-SAFE_POLICY = 0
-ALTERNATIVE_POLICY = 1
-
-REMOVED = 0
-ADDED = 1
-CHANGED = 2
-CHANGED_ADDED = 3
-EVENT_TYPES = (REMOVED, CHANGED, ADDED, CHANGED_ADDED)
-antiEvent = {REMOVED: ADDED,
- ADDED: REMOVED,
- CHANGED: CHANGED,
- CHANGED_ADDED: CHANGED_ADDED,
- }.get
-
-ADDED_EVENTS = (CHANGED, ADDED, CHANGED_ADDED)
-
-
-class CatalogEventQueue(Persistent):
- """Event queue for catalog events
-
- This is a rather odd queue. It organizes events by object, where
- objects are identified by uids, which happen to be string paths.
-
- One way that this queue is extremely odd is that it really only
- keeps track of the last event for an object. This is because we
- really only *care* about the last event for an object.
-
- There are three types of events:
-
- ADDED -- An object was added to the catalog
-
- CHANGED -- An object was changed
-
- REMOVED -- An object was removed from the catalog
-
- CHANGED_ADDED -- Add object was added and subsequently changed.
- This event is a consequence of the queue implementation.
-
- Note that, although we only keep track of the most recent
- event. there are rules for how the most recent event can be
- updated:
-
- - It is illegal to update an ADDED, CHANGED, or CHANGED_ADDED
- event with an ADDED event or
-
- - to update a REMOVED event with a CHANGED event.
-
- We have a problem because applications don't really indicate
- whether they are are adding, or just updating. We deduce add
- events by examining the catalog and event queue states.
-
- Also note that, when events are applied to the catalog, events may
- have no effect.
-
- - If an object is in the catalog, ADDED events are equivalent to
- CHANGED events.
-
- - If an object is not in the catalog, REMOVED and CHANGED events
- have no effect.
-
- If we undo a transaction, we generate an anti-event. The anti
- event of ADDED id REMOVED, of REMOVED is ADDED, and of CHANGED is
- CHANGED.
-
- Note that these rules represent heuristics that attempt to provide
- efficient and sensible behavior for most cases. They are not "correct" in
- that they handle cases that may not seem handleable. For example,
- consider a sequence of transactions:
-
- T1 adds an object
- T2 removes the object
- T3 adds the object
- T4 processes the queue
- T5 undoes T1
-
- It's not clear what should be done in this case? We decide to
- generate a remove event, even though a later transaction added the
- object again. Is this correct? It's hard to say. The decision we
- make is not horrible and it allows us to provide a very efficient
- implementation. See the unit tests for other scenarios. Feel
- free to think of cases for which our decisions are unacceptably
- wrong and write unit tests for these cases.
-
- There are two kinds of transactions that affect the queue:
-
- - Application transactions always add or modify events. They never
- remove events.
-
- - Queue processing transactions always remove events.
-
- """
-
- _conflict_policy = SAFE_POLICY
-
- def __init__(self, conflict_policy=SAFE_POLICY):
-
- # Mapping from uid -> (generation, event type)
- self._data = {}
- self._conflict_policy = conflict_policy
-
- def __nonzero__(self):
- return not not self._data
-
- def __len__(self):
- return len(self._data)
-
- def update(self, uid, etype):
- assert etype in EVENT_TYPES
- data = self._data
- current = data.get(uid)
- if current is not None:
- generation, current = current
- if current in ADDED_EVENTS and etype is ADDED:
- raise TypeError("Attempt to add an object that is already "
- "in the catalog")
- if current is REMOVED and etype is CHANGED:
- raise TypeError("Attempt to change an object that has "
- "been removed")
-
- if ((current is ADDED or current is CHANGED_ADDED)
- and etype is CHANGED):
- etype = CHANGED_ADDED
-
- else:
- generation = 0
-
- data[uid] = generation+1, etype
-
- self._p_changed = 1
-
- def getEvent(self, uid):
- state = self._data.get(uid)
- if state is not None:
- state = state[1]
- return state
-
- def process(self, limit=None):
- """Removes and returns events from this queue.
-
- If limit is specified, at most (limit) events are removed.
- """
- data = self._data
- if not limit or len(data) <= limit:
- self._data = {}
- return data
- else:
- self._p_changed = 1
- res = {}
- keys = data.keys()[:limit]
- for key in keys:
- res[key] = data[key]
- del data[key]
- return res
-
- def _p_resolveConflict(self, oldstate, committed, newstate):
- # Apply the changes made in going from old to newstate to
- # committed
-
- # Note that in the case of undo, the olddata is the data for
- # the transaction being undone and newdata is the data for the
- # transaction previous to the undone transaction.
-
- # Find the conflict policy on the new state to make sure changes
- # to it will be applied
- policy = newstate['_conflict_policy']
-
- # Committed is always the currently committed data.
- oldstate_data = oldstate['_data']
- committed_data = committed['_data']
- newstate_data = newstate['_data']
-
- # Merge newstate changes into committed
- for uid, new in newstate_data.items():
-
- # Decide if this is a change
- old = oldstate_data.get(uid)
- current = committed_data.get(uid)
-
- if new != old:
- # something changed
-
- if old is not None:
- # got a repeat event
- if new[0] < old[0]:
- # This was an undo, so give the event the undo
- # time and convert to an anti event of the old
- # (undone) event.
- new = (0, antiEvent(old[1]))
- elif new[1] is ADDED:
- if policy == SAFE_POLICY:
- logger.error('Queue conflict on %s: ADDED on existing item' % uid)
- raise ConflictError
- else:
- if current and current[1] == REMOVED:
- new = current
- else:
- new = (current[0]+1, CHANGED_ADDED)
-
-
- # remove this event from old, so that we don't
- # mess with it later.
- del oldstate_data[uid]
-
- # Check aqainst current value. Either we want a
- # different event, in which case we give up, or we
- # do nothing.
- if current is not None:
- if current[1] != new[1]:
- if policy == SAFE_POLICY:
- # This is too complicated, bail
- logger.error('Queue conflict on %s' % uid)
- raise ConflictError
- elif REMOVED not in (new[1], current[1]):
- new = (current[0]+1, CHANGED_ADDED)
- committed_data[uid] = new
- elif ( current[0] < new[0] and
- new[1] == REMOVED ):
- committed_data[uid] = new
-
- # remove this event from old, so that we don't
- # mess with it later.
- if oldstate_data.get(uid) is not None:
- del oldstate_data[uid]
-
- # nothing to do
- continue
-
- committed_data[uid] = new
-
- # Now handle remaining events in old that weren't in new.
- # These *must* be undone events!
- for uid, old in oldstate_data.items():
- new = (0, antiEvent(old[1]))
-
- # See above
- current = committed_data.get(uid)
- if current is not None:
- if current[1] != new[1]:
- # This is too complicated, bail
- logger.error('Queue conflict on %s processing undos' % uid)
- raise ConflictError
- # nothing to do
- continue
-
- committed_data[uid] = new
-
- return { '_data': committed_data
- , '_conflict_policy' : policy
- }
-
-__doc__ = CatalogEventQueue.__doc__ + __doc__
-
-
-
-# Old worries
-
-# We have a problem. We have to make sure that we don't lose too
-# much history to undo, but we don't want to retain the entire
-# history. We certainly don't want to execute the entire history
-# when we execute a trans.
-#
-# Baah, no worry, if we undo in a series of unprocessed events, we
-# simply restore the old event, which we have in the old state.
-
-
Deleted: Products.QueueCatalog/trunk/CatalogEventQueueSet.py
===================================================================
--- Products.QueueCatalog/trunk/CatalogEventQueueSet.py 2008-05-13 10:36:31 UTC (rev 86690)
+++ Products.QueueCatalog/trunk/CatalogEventQueueSet.py 2008-05-13 10:37:50 UTC (rev 86691)
@@ -1,221 +0,0 @@
-##############################################################################
-#
-# Copyright (c) 2002-2006 Zope Corporation and Contributors.
-# All Rights Reserved.
-#
-# This software is subject to the provisions of the Zope Public License,
-# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
-# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
-# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
-# FOR A PARTICULAR PURPOSE.
-#
-##############################################################################
-""" Classes: CatalogEventQueueSet
-
-$Id$
-"""
-from __future__ import generators
-
-from Interface import Interface
-
-from OFS.SimpleItem import SimpleItem
-
-from Products.QueueCatalog.CatalogEventQueue import CatalogEventQueue
-from Products.QueueCatalog.CatalogEventQueue import EVENT_TYPES
-from Products.QueueCatalog.CatalogEventQueue import ADDED_EVENTS
-from Products.QueueCatalog.CatalogEventQueue import ADDED
-from Products.QueueCatalog.CatalogEventQueue import CHANGED
-from Products.QueueCatalog.CatalogEventQueue import CHANGED_ADDED
-from Products.QueueCatalog.CatalogEventQueue import REMOVED
-
-class ICatalogEventQueueSetDelegate( Interface ):
-
- """ Interface for the "store" underlying a CEQS.
- """
-
- def hasUID( uid ):
-
- """ Do we already have UID?
- """
-
- def add( uid ):
-
- """ Add UID to the store.
-
- o Delegate is responsible for finding object using UID.
- """
-
- def change( uid ):
-
- """ Update UID within the store.
-
- o Delegate is responsible for finding object using UID.
- """
-
- def remove( uid ):
-
- """ Remove UID from the store.
- """
-
-
-class CatalogEventQueueSet( SimpleItem ):
-
- """ Manage a set of CatalogEventQueue objects.
- """
-
- def __init__( self
- , delegate=None
- , bucket_count=1009
- ):
-
- self.setDelegate( delegate )
- self.setBucketCount( bucket_count )
-
- #
- # Accessors
- #
- def getDelegate( self ):
-
- """ Return the callback object used to process events.
- """
- return self._delegate
-
- def getBucketCount( self ):
-
- """ How many buckets in our hashtable?
- """
- return self._bucket_count
-
- def getEvent( self, uid ):
-
- """ Return the most recent event, if any, for uid.
- """
- return self._getQueue( uid ).getEvent( uid )
-
- def listEvents( self ):
-
- """ Return all events we currently know about.
-
- o Each item in the returned sequence is a tuple, ( uid, event ).
-
- o This function is a generator.
-
- o This function does *not* drain the queues.
- """
- for queue in filter( None, self._queues ):
- for item in queue._data.items():
- uid, ( t, event ) = item
- yield uid, event
-
-
- #
- # Mutators
- #
- def setDelegate( self, delegate ):
-
- """ Update our delegate.
-
- o If not None, 'delegate' must implement ICatalogEventQueueSetDelegate,
- else raise ValueError.
- """
- if ( delegate is not None
- and not ICatalogEventQueueSetDelegate.isImplementedBy( delegate )
- ):
- raise ValueError, "'delegate' doesn't implement ICEQSD!"
-
- self._delegate = delegate
-
- def setBucketCount( self, bucket_count ):
-
- """ Resize the hashtable.
-
- o 'bucket_count' must be a positive, prime integer, else raise
- ValueError.
-
- o N.B.: If successful, we destroy any existing queues!
- """
- if ( type( bucket_count ) is not type( 0 )
- or bucket_count < 3
- or not _isPrime( bucket_count )
- ):
- raise ValueError, 'bucket_count must be a positive, prime int!'
-
- self._bucket_count = bucket_count
- self._clear()
-
- def update( self, uid, event ):
-
- """ Add an event to our queue.
-
- o 'event' must be one of the EVENT_TYPES.
- """
- if event not in EVENT_TYPES:
- raise ValueError, 'Not a known event: %s' % event
-
- self._getQueue( uid ).update( uid, event )
-
- def process( self ):
-
- """ Process events in the queues.
- """
- for queue in filter( None, self._queues ):
- for item in queue.process().items():
-
- if self._delegate is None:
- continue
-
- uid, ( t, event ) = item
-
- if event == ADDED or event == CHANGED_ADDED:
- self._delegate.add( uid )
- elif event == CHANGED:
- self._delegate.change( uid )
- elif event == REMOVED:
- self._delegate.remove( uid )
-
- #
- # Helper methods
- #
- def _getQueue( self, uid ):
-
- return self._queues[ hash( uid ) % self._bucket_count ]
-
- def _clear( self ):
-
- self._queues = [ CatalogEventQueue()
- for i in range( self._bucket_count ) ]
-
-def _isPrime( x ):
-
- """ isPrime(int x) --> boolean
-
- isPrime(x) checks if a given number x is prime. It follows the following
- decision tree:
- if x is 1 it is a special case (ie: not prime),
- if x is either 2 or 3, it is prime,
- if x is even, it is not prime.
- if x is not one of these obvious cases:
- check if x is divisible by the numbers in the range from 3 to
- floor(sqrt(x)).
-
- From: http://jijo.free.net.ph/programming/python/code/prime.py
- """
- if (x == 1): # Special case, neither prime nor composite
- return None
- elif ((x == 2) or (x == 3)):
- return x
- elif ((x % 2) == 0):
- return None
- else:
- flag = 0
- i = 3
- while 1:
- if (x % i) == 0:
- return 0
- elif (i * i) > x:
- return 1
- i += 2
-
-
-
Copied: Products.QueueCatalog/trunk/Products/QueueCatalog/CHANGES.txt (from rev 86689, Products.QueueCatalog/trunk/CHANGES.txt)
===================================================================
--- Products.QueueCatalog/trunk/Products/QueueCatalog/CHANGES.txt (rev 0)
+++ Products.QueueCatalog/trunk/Products/QueueCatalog/CHANGES.txt 2008-05-13 10:37:50 UTC (rev 86691)
@@ -0,0 +1,81 @@
+QueueCatalog Product Changelog
+
+ QueueCatalog 1.5 (2007/12/15)
+
+ - Replaced validateValue call with proper validate call. This
+ makes QueueCatalog compatible with Zope 2.9
+
+ QueueCatalog 1.4 (2006/05/09)
+
+ - Added a second catalog queue conflict resolution strategy and made
+ the strategy configurable via the 'Configure' ZMI tab. The new policy
+ can be tried when ZODB conflict errors emanating from the catalog
+ queue buckets become a problem. It attempts to avoid conflicts by
+ doing even more guessing about what the behavior should be, rather
+ than deeming a conflicting situation as too insane to handle and
+ raising a ConflictError.
+
+ - Removed all uses of zLOG in favor of the Python logging module
+
+ - Updated some help screens
+
+ - Added a dependencies document
+
+ - Updated the tests to run on Zope 2.8+
+
+ - Factored out queue conflict testing after changing it to use a quick
+ and dirty FileStorage, and expanding it a little.
+
+ - Removed the security declaration for a method named refreshCatalog
+ which was added in version 1.0 - this is the wrong place to apply
+ product-specific settings for a product the QueueCatalog is not
+ dependent on.
+
+ - CVS-only, tag 'QueueCatalog-post-1_3-error-logging'
+
+ - When performing deferred indexing, don't attempt (again) to
+ assign brains metadata
+
+ - When indexing an object raises an Exception, log it, and remove
+ the object from the queue
+
+ QueueCatalog 1.3 (2004/11/22)
+
+ - Released by ctheune at:
+ http://zope.org/Members/ctheune/QueueCatalog/1.3/QueueCatalog-1.3.tar.gz
+
+ - CVS tag 'QueueCatalog-1_3'
+
+ - QueueCatalog.py: Made the immediate processing of DELETE-Requests
+ more efficient. Christian Theune <ct at gocept.com>
+
+ - QueueCatalog.py: Proxied ZCatalog.Indexes attribute.
+
+ - QueueCatalog.py: Corrected a permission declaration typo for
+ 'manage_size'.
+
+ - QueueCatalog.py: Modified 'manage_queue' to display a list of queued
+ items (limit 100 items).
+
+ QueueCatalog 1.2 (2003/11/06)
+
+ - CVS-only release, tag 'QueueCatalog-1_2'
+
+ - Add uidForObject hook which allows the underlying catalog to specify
+ uids for new objects being indexed. If this hook is not implemented,
+ uids are abs paths as before.
+
+ QueueCatalog 1.1 (2003/08/06)
+
+ - CVS-only release, tag 'QueueCatalog-1_1'
+
+ - Fix brown-paper bag error which prevented queue processing from
+ *ever* working.
+
+ QueueCatalog 1.0 (2003/06/25)
+
+ - CVS-only release, tag 'QueueCatalog-1_0'
+
+ - QueueCatalog.py (QueueCatalog.refreshCatalog): Added
+ refreshCatalog method that queues the reindexing. Plone uses this
+ method on portal creation. Sidnei da Silva <sidnei at x3ng.com>
Copied: Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueue.py (from rev 86689, Products.QueueCatalog/trunk/CatalogEventQueue.py)
===================================================================
--- Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueue.py (rev 0)
+++ Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueue.py 2008-05-13 10:37:50 UTC (rev 86691)
@@ -0,0 +1,289 @@
+##############################################################################
+#
+# Copyright (c) 2002-2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""
+$Id$
+"""
+
+import logging
+
+from Persistence import Persistent
+from ZODB.POSException import ConflictError
+
+logger = logging.getLogger('event.QueueCatalog')
+
+SAFE_POLICY = 0
+ALTERNATIVE_POLICY = 1
+
+REMOVED = 0
+ADDED = 1
+CHANGED = 2
+CHANGED_ADDED = 3
+EVENT_TYPES = (REMOVED, CHANGED, ADDED, CHANGED_ADDED)
+antiEvent = {REMOVED: ADDED,
+ ADDED: REMOVED,
+ CHANGED: CHANGED,
+ CHANGED_ADDED: CHANGED_ADDED,
+ }.get
+
+ADDED_EVENTS = (CHANGED, ADDED, CHANGED_ADDED)
+
+
+class CatalogEventQueue(Persistent):
+ """Event queue for catalog events
+
+ This is a rather odd queue. It organizes events by object, where
+ objects are identified by uids, which happen to be string paths.
+
+ One way that this queue is extremely odd is that it really only
+ keeps track of the last event for an object. This is because we
+ really only *care* about the last event for an object.
+
+ There are three types of events:
+
+ ADDED -- An object was added to the catalog
+
+ CHANGED -- An object was changed
+
+ REMOVED -- An object was removed from the catalog
+
+ CHANGED_ADDED -- Add object was added and subsequently changed.
+ This event is a consequence of the queue implementation.
+
+ Note that, although we only keep track of the most recent
+ event. there are rules for how the most recent event can be
+ updated:
+
+ - It is illegal to update an ADDED, CHANGED, or CHANGED_ADDED
+ event with an ADDED event or
+
+ - to update a REMOVED event with a CHANGED event.
+
+ We have a problem because applications don't really indicate
+ whether they are are adding, or just updating. We deduce add
+ events by examining the catalog and event queue states.
+
+ Also note that, when events are applied to the catalog, events may
+ have no effect.
+
+ - If an object is in the catalog, ADDED events are equivalent to
+ CHANGED events.
+
+ - If an object is not in the catalog, REMOVED and CHANGED events
+ have no effect.
+
+ If we undo a transaction, we generate an anti-event. The anti
+ event of ADDED id REMOVED, of REMOVED is ADDED, and of CHANGED is
+ CHANGED.
+
+ Note that these rules represent heuristics that attempt to provide
+ efficient and sensible behavior for most cases. They are not "correct" in
+ that they handle cases that may not seem handleable. For example,
+ consider a sequence of transactions:
+
+ T1 adds an object
+ T2 removes the object
+ T3 adds the object
+ T4 processes the queue
+ T5 undoes T1
+
+ It's not clear what should be done in this case? We decide to
+ generate a remove event, even though a later transaction added the
+ object again. Is this correct? It's hard to say. The decision we
+ make is not horrible and it allows us to provide a very efficient
+ implementation. See the unit tests for other scenarios. Feel
+ free to think of cases for which our decisions are unacceptably
+ wrong and write unit tests for these cases.
+
+ There are two kinds of transactions that affect the queue:
+
+ - Application transactions always add or modify events. They never
+ remove events.
+
+ - Queue processing transactions always remove events.
+
+ """
+
+ _conflict_policy = SAFE_POLICY
+
+ def __init__(self, conflict_policy=SAFE_POLICY):
+
+ # Mapping from uid -> (generation, event type)
+ self._data = {}
+ self._conflict_policy = conflict_policy
+
+ def __nonzero__(self):
+ return not not self._data
+
+ def __len__(self):
+ return len(self._data)
+
+ def update(self, uid, etype):
+ assert etype in EVENT_TYPES
+ data = self._data
+ current = data.get(uid)
+ if current is not None:
+ generation, current = current
+ if current in ADDED_EVENTS and etype is ADDED:
+ raise TypeError("Attempt to add an object that is already "
+ "in the catalog")
+ if current is REMOVED and etype is CHANGED:
+ raise TypeError("Attempt to change an object that has "
+ "been removed")
+
+ if ((current is ADDED or current is CHANGED_ADDED)
+ and etype is CHANGED):
+ etype = CHANGED_ADDED
+
+ else:
+ generation = 0
+
+ data[uid] = generation+1, etype
+
+ self._p_changed = 1
+
+ def getEvent(self, uid):
+ state = self._data.get(uid)
+ if state is not None:
+ state = state[1]
+ return state
+
+ def process(self, limit=None):
+ """Removes and returns events from this queue.
+
+ If limit is specified, at most (limit) events are removed.
+ """
+ data = self._data
+ if not limit or len(data) <= limit:
+ self._data = {}
+ return data
+ else:
+ self._p_changed = 1
+ res = {}
+ keys = data.keys()[:limit]
+ for key in keys:
+ res[key] = data[key]
+ del data[key]
+ return res
+
+ def _p_resolveConflict(self, oldstate, committed, newstate):
+ # Apply the changes made in going from old to newstate to
+ # committed
+
+ # Note that in the case of undo, the olddata is the data for
+ # the transaction being undone and newdata is the data for the
+ # transaction previous to the undone transaction.
+
+ # Find the conflict policy on the new state to make sure changes
+ # to it will be applied
+ policy = newstate['_conflict_policy']
+
+ # Committed is always the currently committed data.
+ oldstate_data = oldstate['_data']
+ committed_data = committed['_data']
+ newstate_data = newstate['_data']
+
+ # Merge newstate changes into committed
+ for uid, new in newstate_data.items():
+
+ # Decide if this is a change
+ old = oldstate_data.get(uid)
+ current = committed_data.get(uid)
+
+ if new != old:
+ # something changed
+
+ if old is not None:
+ # got a repeat event
+ if new[0] < old[0]:
+ # This was an undo, so give the event the undo
+ # time and convert to an anti event of the old
+ # (undone) event.
+ new = (0, antiEvent(old[1]))
+ elif new[1] is ADDED:
+ if policy == SAFE_POLICY:
+ logger.error('Queue conflict on %s: ADDED on existing item' % uid)
+ raise ConflictError
+ else:
+ if current and current[1] == REMOVED:
+ new = current
+ else:
+ new = (current[0]+1, CHANGED_ADDED)
+
+
+ # remove this event from old, so that we don't
+ # mess with it later.
+ del oldstate_data[uid]
+
+ # Check aqainst current value. Either we want a
+ # different event, in which case we give up, or we
+ # do nothing.
+ if current is not None:
+ if current[1] != new[1]:
+ if policy == SAFE_POLICY:
+ # This is too complicated, bail
+ logger.error('Queue conflict on %s' % uid)
+ raise ConflictError
+ elif REMOVED not in (new[1], current[1]):
+ new = (current[0]+1, CHANGED_ADDED)
+ committed_data[uid] = new
+ elif ( current[0] < new[0] and
+ new[1] == REMOVED ):
+ committed_data[uid] = new
+
+ # remove this event from old, so that we don't
+ # mess with it later.
+ if oldstate_data.get(uid) is not None:
+ del oldstate_data[uid]
+
+ # nothing to do
+ continue
+
+ committed_data[uid] = new
+
+ # Now handle remaining events in old that weren't in new.
+ # These *must* be undone events!
+ for uid, old in oldstate_data.items():
+ new = (0, antiEvent(old[1]))
+
+ # See above
+ current = committed_data.get(uid)
+ if current is not None:
+ if current[1] != new[1]:
+ # This is too complicated, bail
+ logger.error('Queue conflict on %s processing undos' % uid)
+ raise ConflictError
+ # nothing to do
+ continue
+
+ committed_data[uid] = new
+
+ return { '_data': committed_data
+ , '_conflict_policy' : policy
+ }
+
+__doc__ = CatalogEventQueue.__doc__ + __doc__
+
+
+
+# Old worries
+
+# We have a problem. We have to make sure that we don't lose too
+# much history to undo, but we don't want to retain the entire
+# history. We certainly don't want to execute the entire history
+# when we execute a trans.
+#
+# Baah, no worry, if we undo in a series of unprocessed events, we
+# simply restore the old event, which we have in the old state.
+
+
Copied: Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueueSet.py (from rev 86689, Products.QueueCatalog/trunk/CatalogEventQueueSet.py)
===================================================================
--- Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueueSet.py (rev 0)
+++ Products.QueueCatalog/trunk/Products/QueueCatalog/CatalogEventQueueSet.py 2008-05-13 10:37:50 UTC (rev 86691)
@@ -0,0 +1,221 @@
+##############################################################################
+#
+# Copyright (c) 2002-2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+""" Classes: CatalogEventQueueSet
+
+$Id$
+"""
+from __future__ import generators
+
+from Interface import Interface
+
+from OFS.SimpleItem import SimpleItem
+
+from Products.QueueCatalog.CatalogEventQueue import CatalogEventQueue
+from Products.QueueCatalog.CatalogEventQueue import EVENT_TYPES
+from Products.QueueCatalog.CatalogEventQueue import ADDED_EVENTS
+from Products.QueueCatalog.CatalogEventQueue import ADDED
+from Products.QueueCatalog.CatalogEventQueue import CHANGED
+from Products.QueueCatalog.CatalogEventQueue import CHANGED_ADDED
+from Products.QueueCatalog.CatalogEventQueue import REMOVED
+
+class ICatalogEventQueueSetDelegate( Interface ):
+
+ """ Interface for the "store" underlying a CEQS.
+ """
+
+ def hasUID( uid ):
+
+ """ Do we already have UID?
+ """
+
+ def add( uid ):
+
+ """ Add UID to the store.
+
+ o Delegate is responsible for finding object using UID.
+ """
+
+ def change( uid ):
+
+ """ Update UID within the store.
+
+ o Delegate is responsible for finding object using UID.
+ """
+
+ def remove( uid ):
+
+ """ Remove UID from the store.
+ """
+
+
+class CatalogEventQueueSet( SimpleItem ):
+
+ """ Manage a set of CatalogEventQueue objects.
+ """
+
+ def __init__( self
+ , delegate=None
+ , bucket_count=1009
+ ):
+
+ self.setDelegate( delegate )
+ self.setBucketCount( bucket_count )
+
+ #
+ # Accessors
+ #
+ def getDelegate( self ):
+
+ """ Return the callback object used to process events.
+ """
+ return self._delegate
+
+ def getBucketCount( self ):
+
+ """ How many buckets in our hashtable?
+ """
+ return self._bucket_count
+
+ def getEvent( self, uid ):
+
+ """ Return the most recent event, if any, for uid.
+ """
+ return self._getQueue( uid ).getEvent( uid )
+
+ def listEvents( self ):
+
+ """ Return all events we currently know about.
+
+ o Each item in the returned sequence is a tuple, ( uid, event ).
+
+ o This function is a generator.
+
+ o This function does *not* drain the queues.
+ """
+ for queue in filter( None, self._queues ):
+ for item in queue._data.items():
+ uid, ( t, event ) = item
+ yield uid, event
+
+
+ #
+ # Mutators
+ #
+ def setDelegate( self, delegate ):
+
+ """ Update our delegate.
+
+ o If not None, 'delegate' must implement ICatalogEventQueueSetDelegate,
+ else raise ValueError.
+ """
+ if ( delegate is not None
+ and not ICatalogEventQueueSetDelegate.isImplementedBy( delegate )
+ ):
+ raise ValueError, "'delegate' doesn't implement ICEQSD!"
+
+ self._delegate = delegate
+
+ def setBucketCount( self, bucket_count ):
+
+ """ Resize the hashtable.
+
+ o 'bucket_count' must be a positive, prime integer, else raise
+ ValueError.
+
+ o N.B.: If successful, we destroy any existing queues!
+ """
+ if ( type( bucket_count ) is not type( 0 )
+ or bucket_count < 3
+ or not _isPrime( bucket_count )
+ ):
+ raise ValueError, 'bucket_count must be a positive, prime int!'
+
+ self._bucket_count = bucket_count
+ self._clear()
+
+ def update( self, uid, event ):
+
+ """ Add an event to our queue.
+
+ o 'event' must be one of the EVENT_TYPES.
+ """
+ if event not in EVENT_TYPES:
+ raise ValueError, 'Not a known event: %s' % event
+
+ self._getQueue( uid ).update( uid, event )
+
+ def process( self ):
+
+ """ Process events in the queues.
+ """
+ for queue in filter( None, self._queues ):
+ for item in queue.process().items():
+
+ if self._delegate is None:
+ continue
+
+ uid, ( t, event ) = item
+
+ if event == ADDED or event == CHANGED_ADDED:
+ self._delegate.add( uid )
+ elif event == CHANGED:
+ self._delegate.change( uid )
+ elif event == REMOVED:
+ self._delegate.remove( uid )
+
+ #
+ # Helper methods
+ #
+ def _getQueue( self, uid ):
+
+ return self._queues[ hash( uid ) % self._bucket_count ]
+
+ def _clear( self ):
+
+ self._queues = [ CatalogEventQueue()
+ for i in range( self._bucket_count ) ]
+
+def _isPrime( x ):
+
+ """ isPrime(int x) --> boolean
+
+ isPrime(x) checks if a given number x is prime. It follows the following
+ decision tree:
+ if x is 1 it is a special case (ie: not prime),
+ if x is either 2 or 3, it is prime,
+ if x is even, it is not prime.
+ if x is not one of these obvious cases:
+ check if x is divisible by the numbers in the range from 3 to
+ floor(sqrt(x)).
+
+ From: http://jijo.free.net.ph/programming/python/code/prime.py
+ """
+ if (x == 1): # Special case, neither prime nor composite
+ return None
+ elif ((x == 2) or (x == 3)):
+ return x
+ elif ((x % 2) == 0):
+ return None
+ else:
+ flag = 0
+ i = 3
+ while 1:
+ if (x % i) == 0:
+ return 0
+ elif (i * i) > x:
+ return 1
+ i += 2
+
+
+
Copied: Products.QueueCatalog/trunk/Products/QueueCatalog/dtml (from rev 86689, Products.QueueCatalog/trunk/dtml)
Copied: Products.QueueCatalog/trunk/Products/QueueCatalog/help (from rev 86689, Products.QueueCatalog/trunk/help)
Copied: Products.QueueCatalog/trunk/Products/QueueCatalog/version.txt (from rev 86689, Products.QueueCatalog/trunk/version.txt)
===================================================================
--- Products.QueueCatalog/trunk/Products/QueueCatalog/version.txt (rev 0)
+++ Products.QueueCatalog/trunk/Products/QueueCatalog/version.txt 2008-05-13 10:37:50 UTC (rev 86691)
@@ -0,0 +1 @@
+1.5
Copied: Products.QueueCatalog/trunk/Products/QueueCatalog/www (from rev 86689, Products.QueueCatalog/trunk/www)
Deleted: Products.QueueCatalog/trunk/version.txt
===================================================================
--- Products.QueueCatalog/trunk/version.txt 2008-05-13 10:36:31 UTC (rev 86690)
+++ Products.QueueCatalog/trunk/version.txt 2008-05-13 10:37:50 UTC (rev 86691)
@@ -1 +0,0 @@
-1.5
More information about the Checkins
mailing list