[Checkins] SVN: zc.catalogqueue/trunk/s add some processing history
information (lastProcessedTime, totalProcessed)
Fred L. Drake, Jr.
fdrake at gmail.com
Fri May 9 10:34:33 EDT 2008
Log message for revision 86571:
add some processing history information (lastProcessedTime, totalProcessed)
Changed:
U zc.catalogqueue/trunk/setup.py
U zc.catalogqueue/trunk/src/zc/catalogqueue/CHANGES.txt
U zc.catalogqueue/trunk/src/zc/catalogqueue/interfaces.py
U zc.catalogqueue/trunk/src/zc/catalogqueue/queue.py
U zc.catalogqueue/trunk/src/zc/catalogqueue/queue.txt
-=-
Modified: zc.catalogqueue/trunk/setup.py
===================================================================
--- zc.catalogqueue/trunk/setup.py 2008-05-09 14:10:36 UTC (rev 86570)
+++ zc.catalogqueue/trunk/setup.py 2008-05-09 14:34:33 UTC (rev 86571)
@@ -46,8 +46,10 @@
namespace_packages = ['zc'],
package_dir = {'': 'src'},
install_requires = [
+ 'pytz',
'setuptools',
'ZODB3',
+ 'zope.interface',
],
zip_safe = False,
entry_points=entry_points,
Modified: zc.catalogqueue/trunk/src/zc/catalogqueue/CHANGES.txt
===================================================================
--- zc.catalogqueue/trunk/src/zc/catalogqueue/CHANGES.txt 2008-05-09 14:10:36 UTC (rev 86570)
+++ zc.catalogqueue/trunk/src/zc/catalogqueue/CHANGES.txt 2008-05-09 14:34:33 UTC (rev 86571)
@@ -1,6 +1,18 @@
Change History
**************
+0.2 (unreleased)
+================
+
+Added some processing state information as attributes of the queue object:
+
+``lastProcessedTime``
+ Time of the last successful processing.
+
+``totalProcessed``
+ Total number of cataloging events processed.
+
+
0.1 (2008-04-16)
================
Modified: zc.catalogqueue/trunk/src/zc/catalogqueue/interfaces.py
===================================================================
--- zc.catalogqueue/trunk/src/zc/catalogqueue/interfaces.py 2008-05-09 14:10:36 UTC (rev 86570)
+++ zc.catalogqueue/trunk/src/zc/catalogqueue/interfaces.py 2008-05-09 14:34:33 UTC (rev 86571)
@@ -36,4 +36,22 @@
Catalogs is a multi-iterable collection of
zope.index.interfaces.IInjection objects to be updated.
"""
-
+
+ lastProcessedTime = zope.interface.Attribute(
+ """Time the queue was last processed.
+
+ This represents the time when the last successful call to `process()`
+ returned. If this is ``None``, the queue has not been processed since
+ this field was added. This may simply mean that the queue was last
+ processed before the implementation supported tracking this.
+
+ """)
+
+ totalProcessed = zope.interface.Attribute(
+ """Number of cataloging events processed.
+
+ A value of 0 may be caused by the last processing having happened
+ before this information was tracked. If ``lastProcessedTime`` is set,
+ this value may be considered valid, even if 0.
+
+ """)
Modified: zc.catalogqueue/trunk/src/zc/catalogqueue/queue.py
===================================================================
--- zc.catalogqueue/trunk/src/zc/catalogqueue/queue.py 2008-05-09 14:10:36 UTC (rev 86570)
+++ zc.catalogqueue/trunk/src/zc/catalogqueue/queue.py 2008-05-09 14:34:33 UTC (rev 86571)
@@ -14,18 +14,24 @@
from zc.catalogqueue.CatalogEventQueue import REMOVED, CHANGED, ADDED
+import datetime
import logging
import persistent
+import pytz
import zc.catalogqueue.CatalogEventQueue
import zc.catalogqueue.interfaces
import zope.interface
logger = logging.getLogger(__name__)
+
class CatalogQueue(persistent.Persistent):
zope.interface.implements(zc.catalogqueue.interfaces.ICatalogQueue)
+ lastProcessedTime = None
+ totalProcessed = 0
+
_buckets = 1009 # Maybe configurable someday
def __init__(self):
@@ -65,4 +71,7 @@
if done >= limit:
break
+ self.lastProcessedTime = datetime.datetime.now(pytz.UTC)
+ self.totalProcessed += done
+
return done
Modified: zc.catalogqueue/trunk/src/zc/catalogqueue/queue.txt
===================================================================
--- zc.catalogqueue/trunk/src/zc/catalogqueue/queue.txt 2008-05-09 14:10:36 UTC (rev 86570)
+++ zc.catalogqueue/trunk/src/zc/catalogqueue/queue.txt 2008-05-09 14:34:33 UTC (rev 86571)
@@ -15,6 +15,16 @@
[<InterfaceClass zc.catalogqueue.interfaces.ICatalogQueue>,
<InterfaceClass persistent.interfaces.IPersistent>]
+There are some bits of information that the queue maintains regarding its own
+processing state. The time of last processing and the total number of
+cataloging events processed are available. Since this queue hasn't been
+processed yet, these have some initial values:
+
+ >>> print queue.lastProcessedTime
+ None
+ >>> queue.totalProcessed
+ 0
+
Queues are used in 2 ways. As content are modified, we call add,
update, and remove methods on the queue:
@@ -45,6 +55,14 @@
>>> queue.update(0)
>>> queue.update(0)
+At this point, we've added several events, but haven't processed the queue, so
+we expect the ``lastProcessedTime`` and ``totalProcessed`` to be unchanged:
+
+ >>> print queue.lastProcessedTime
+ None
+ >>> queue.totalProcessed
+ 0
+
Periodically, we call process on the queue. We need to pass an ids
object and a collection of injection (catalog) objects:
@@ -87,12 +105,32 @@
- The number of objects processed is returned.
+The processing information has been updated on the queue:
+
+ >>> queue.lastProcessedTime # doctest: +ELLIPSIS
+ datetime.datetime(... tzinfo=<UTC>)
+ >>> queue.totalProcessed
+ 6
+
+ >>> previous_time = queue.lastProcessedTime
+
If we process the queue without additional events, we'll just get 0
back:
>>> queue.process(Ids(), [Injection('cat1'), Injection('cat2')], 10)
0
+The historical processing information is updated:
+
+ >>> queue.lastProcessedTime # doctest: +ELLIPSIS
+ datetime.datetime(... tzinfo=<UTC>)
+ >>> queue.lastProcessedTime > previous_time
+ True
+ >>> queue.totalProcessed
+ 6
+
+ >>> previous_time = queue.lastProcessedTime
+
Of course, the limit argument limits how many events we process:
>>> for i in range(10):
@@ -103,6 +141,8 @@
cat1 indexing 3 object 3
cat1 indexing 4 object 4
5
+ >>> queue.totalProcessed
+ 11
>>> queue.process(Ids(), [Injection('cat1')], 5)
cat1 indexing 5 object 5
@@ -111,6 +151,8 @@
cat1 indexing 8 object 8
cat1 indexing 9 object 9
5
+ >>> queue.totalProcessed
+ 16
(Remember that 0 isn't processed because it can't be found.)
More information about the Checkins
mailing list