[Checkins] SVN: zc.async/trunk/ fixes for monitor tests;
notes of other needed changes
Gary Poster
gary at zope.com
Thu May 8 23:34:04 EDT 2008
Log message for revision 86555:
fixes for monitor tests; notes of other needed changes
Changed:
_U zc.async/trunk/
U zc.async/trunk/setup.py
U zc.async/trunk/src/zc/async/CHANGES.txt
U zc.async/trunk/src/zc/async/README.txt
U zc.async/trunk/src/zc/async/TODO.txt
U zc.async/trunk/src/zc/async/dispatcher.py
U zc.async/trunk/src/zc/async/dispatcher.txt
U zc.async/trunk/src/zc/async/monitor.py
U zc.async/trunk/src/zc/async/monitor.txt
-=-
Property changes on: zc.async/trunk
___________________________________________________________________
Name: svn:ignore
- develop-eggs
bin
parts
.installed.cfg
dist
TEST_THIS_REST_BEFORE_REGISTERING.txt
+ develop-eggs
bin
parts
.installed.cfg
dist
TEST_THIS_REST_BEFORE_REGISTERING.txt
*.kpf
Modified: zc.async/trunk/setup.py
===================================================================
--- zc.async/trunk/setup.py 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/setup.py 2008-05-09 03:33:59 UTC (rev 86555)
@@ -99,7 +99,7 @@
'Twisted>=8.0.1', # 8.0 was setuptools compatible, 8.0.1 had bugfixes.
# note that Twisted builds with warnings, at least with py2.4. It
# seems to still build ok.
- 'zope.bforest>=1.1.1',
+ 'zope.bforest>=1.2',
'zope.component',
'zope.event',
'zope.i18nmessageid',
Modified: zc.async/trunk/src/zc/async/CHANGES.txt
===================================================================
--- zc.async/trunk/src/zc/async/CHANGES.txt 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/src/zc/async/CHANGES.txt 2008-05-09 03:33:59 UTC (rev 86555)
@@ -1,3 +1,23 @@
+1.2 (unreleased)
+================
+
+- more README tweaks.
+
+- converted all reports from the dispatcher, including the monitor output,
+ to use "unpacked" integer oids. This addresses a problem that simplejson
+ was having in trying to interpret the packed string blobs as unicode, and
+ then making zc.ngi fall over.
+
+- added several more tests for the monitor code.
+
+- made the ``async jobs`` monitor command be "up to the minute". Before, it
+ included all of the new and active jobs from the previous poll; now, it
+ also filters out those that have since completed.
+
+- The ``async job`` command was broken, as revealed by a new monitor test.
+ Fixed, which also means we need a new version of zope.bforest (1.2) for a new
+ feature there.
+
1.1 (2008-04-24)
================
Modified: zc.async/trunk/src/zc/async/README.txt
===================================================================
--- zc.async/trunk/src/zc/async/README.txt 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/src/zc/async/README.txt 2008-05-09 03:33:59 UTC (rev 86555)
@@ -493,6 +493,11 @@
a connection off of ``_p_jar``, to get the queue into which the job
was put, or other uses.
+``zc.async.local.getQueue()``
+ The ``getQueue`` function can be used to examine the queue, to put another
+ task into the queue, or other uses. It is sugar for
+ ``zc.async.local.getJob().queue``.
+
``zc.async.local.setLiveAnnotation(name, value, job=None)``
The ``setLiveAnnotation`` tells the agent to set an annotation on a job,
by default the current job, *in another connection*. This makes it
@@ -530,9 +535,9 @@
be used to analyze its non-persistent poll data structure, for instance
(described later in configuration discussions).
-Let's give the first three a whirl. We will write a function that
-examines the job's state while it is being called, and sets the state in
-an annotation, then waits for our flag to finish.
+Let's give three of those a whirl. We will write a function that examines the
+job's state while it is being called, and sets the state in an annotation, then
+waits for our flag to finish.
>>> def annotateStatus():
... zc.async.local.setLiveAnnotation(
@@ -1113,12 +1118,12 @@
>>> import pprint
>>> pprint.pprint(dispatcher.getStatistics()) # doctest: +ELLIPSIS
{'failed': 2,
- 'longest active': ('\x00...', 'unnamed'),
- 'longest failed': ('\x00...', 'unnamed'),
- 'longest successful': ('\x00...', 'unnamed'),
- 'shortest active': ('\x00\...', 'unnamed'),
- 'shortest failed': ('\x00\...', 'unnamed'),
- 'shortest successful': ('\x00...', 'unnamed'),
+ 'longest active': (..., 'unnamed'),
+ 'longest failed': (..., 'unnamed'),
+ 'longest successful': (..., 'unnamed'),
+ 'shortest active': (..., 'unnamed'),
+ 'shortest failed': (..., 'unnamed'),
+ 'shortest successful': (..., 'unnamed'),
'started': 12,
'statistics end': datetime.datetime(2006, 8, 10, 15, 44, 22, 211),
'statistics start': datetime.datetime(2006, 8, 10, 15, 56, 47, 211),
@@ -1151,11 +1156,11 @@
>>> pprint.pprint(dispatcher.getStatistics()) # doctest: +ELLIPSIS
{'failed': 2,
'longest active': None,
- 'longest failed': ('\x00...', 'unnamed'),
- 'longest successful': ('\x00...', 'unnamed'),
+ 'longest failed': (..., 'unnamed'),
+ 'longest successful': (..., 'unnamed'),
'shortest active': None,
- 'shortest failed': ('\x00\...', 'unnamed'),
- 'shortest successful': ('\x00...', 'unnamed'),
+ 'shortest failed': (..., 'unnamed'),
+ 'shortest successful': (..., 'unnamed'),
'started': 10,
'statistics end': datetime.datetime(2006, 8, 10, 15, 46, 52, 211),
'statistics start': datetime.datetime(2006, 8, 10, 15, 56, 52, 211),
@@ -1169,12 +1174,10 @@
only keeps 10 to 12.5 minutes worth of poll information in memory. For
the rest, keep logs and look at them (...and rotate them!).
- The ``getActiveJobIds`` list shows the new task--which is completed, but
- not as of the last poll, so it's still in the list.
+ The ``getActiveJobIds`` list is empty now.
- >>> job_ids = dispatcher.getActiveJobIds()
- >>> len(job_ids)
- 1
+ >>> dispatcher.getActiveJobIds()
+ []
>>> info = dispatcher.getJobInfo(*job_ids[0])
>>> pprint.pprint(info) # doctest: +ELLIPSIS
{'call': "<zc.async.job.Job (oid ..., db 'unnamed') ``zc.async.doctest_test.annotateStatus()``>",
@@ -1202,11 +1205,11 @@
>>> pprint.pprint(dispatcher.getStatistics()) # doctest: +ELLIPSIS
{'failed': 2,
'longest active': None,
- 'longest failed': ('\x00...', 'unnamed'),
- 'longest successful': ('\x00...', 'unnamed'),
+ 'longest failed': (..., 'unnamed'),
+ 'longest successful': (..., 'unnamed'),
'shortest active': None,
- 'shortest failed': ('\x00\...', 'unnamed'),
- 'shortest successful': ('\x00...', 'unnamed'),
+ 'shortest failed': (..., 'unnamed'),
+ 'shortest successful': (..., 'unnamed'),
'started': 22,
'statistics end': datetime.datetime(2006, 8, 10, 15, 46, 52, 211),
'statistics start': datetime.datetime(2006, 8, 10, 15, 57, 47, 211),
Modified: zc.async/trunk/src/zc/async/TODO.txt
===================================================================
--- zc.async/trunk/src/zc/async/TODO.txt 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/src/zc/async/TODO.txt 2008-05-09 03:33:59 UTC (rev 86555)
@@ -1,3 +1,168 @@
+Bugs and improvements:
+
+- in failing a task, jobs are not in agent:
+ <zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``> failed with traceback:
+ Failure: zc.async.interfaces.AbortedError:
+
+ <zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``> failed with traceback:
+ *--- Failure #7 (pickled) ---
+ /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:299: _call_with_retry(...)
+ [ Locals ]
+ res : 'None'
+ self : "<zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``>"
+ tm : '<transaction._manager.ThreadTransactionManager object at 0xb7a5ea4c>'
+ call : '<function <lambda> at 0xb686fd4c>'
+ ct : '0'
+ ( Globals )
+ success_or_failure : '<function success_or_failure at 0xb6cb917c>'
+ __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
+ persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
+ zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
+ pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
+ __name__ : "'zc.async.job'"
+ datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
+ types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
+ transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
+ completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
+ twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
+ _repr : '<function _repr at 0xb6cb9144>'
+ Job : "<class 'zc.async.job.Job'>"
+ ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
+ rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
+ zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
+ BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
+ __doc__ : 'None'
+ /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:291: <lambda>(...)
+ [ Locals ]
+ self : "<zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``>"
+ effective_args : '[]'
+ effective_kwargs : '{}'
+ ( Globals )
+ success_or_failure : '<function success_or_failure at 0xb6cb917c>'
+ __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
+ persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
+ zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
+ pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
+ __name__ : "'zc.async.job'"
+ datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
+ types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
+ transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
+ completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
+ twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
+ _repr : '<function _repr at 0xb6cb9144>'
+ Job : "<class 'zc.async.job.Job'>"
+ ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
+ rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
+ zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
+ BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
+ __doc__ : 'None'
+ /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:362: fail(...)
+ [ Locals ]
+ self : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
+ e : '<zc.async.interfaces.AbortedError instance at 0xb07fa7ac>'
+ ( Globals )
+ success_or_failure : '<function success_or_failure at 0xb6cb917c>'
+ __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
+ persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
+ zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
+ pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
+ __name__ : "'zc.async.job'"
+ datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
+ types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
+ transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
+ completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
+ twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
+ _repr : '<function _repr at 0xb6cb9144>'
+ Job : "<class 'zc.async.job.Job'>"
+ ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
+ rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
+ zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
+ BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
+ __doc__ : 'None'
+ /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:396: resumeCallbacks(...)
+ [ Locals ]
+ callbacks : '[]'
+ length : '1'
+ tm : '<transaction._manager.ThreadTransactionManager object at 0xb7a5ea4c>'
+ self : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
+ j : "<zc.async.job.Job (oid 33530088, db '') ``zc.z4m.content.query.catalog.addProcessingTask(zc.async.queue.Queue (oid 33148381, db ''), delay=datetime.timedelta(0, 600))``>"
+ ( Globals )
+ success_or_failure : '<function success_or_failure at 0xb6cb917c>'
+ __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
+ persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
+ zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
+ pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
+ __name__ : "'zc.async.job'"
+ datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
+ types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
+ transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
+ completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
+ twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
+ _repr : '<function _repr at 0xb6cb9144>'
+ Job : "<class 'zc.async.job.Job'>"
+ ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
+ rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
+ zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
+ BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
+ __doc__ : 'None'
+ /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.py:77: jobCompleted(...)
+ [ Locals ]
+ job : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
+ self : '<zc.async.agent.Agent object at 0xb07e0d2c>'
+ ( Globals )
+ __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.pyc'"
+ persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
+ Agent : "<class 'zc.async.agent.Agent'>"
+ datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
+ chooseFirst : '<function chooseFirst at 0xb6700d4c>'
+ addMainAgentActivationHandler : '<function addMainAgentActivationHandler at 0xb6700dbc>'
+ zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
+ __name__ : "'zc.async.agent'"
+ zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
+ __doc__ : 'None'
+ /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.py:61: remove(...)
+ [ Locals ]
+ item : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
+ self : '<zc.async.agent.Agent object at 0xb07e0d2c>'
+ ( Globals )
+ __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.pyc'"
+ persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
+ Agent : "<class 'zc.async.agent.Agent'>"
+ datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
+ chooseFirst : '<function chooseFirst at 0xb6700d4c>'
+ addMainAgentActivationHandler : '<function addMainAgentActivationHandler at 0xb6700dbc>'
+ zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
+ __name__ : "'zc.async.agent'"
+ zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
+ __doc__ : 'None'
+ /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.py:58: index(...)
+ [ Locals ]
+ i : "<zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``>"
+ ix : '0'
+ self : '<zc.async.agent.Agent object at 0xb07e0d2c>'
+ item : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
+ ( Globals )
+ __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.pyc'"
+ persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
+ Agent : "<class 'zc.async.agent.Agent'>"
+ datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
+ chooseFirst : '<function chooseFirst at 0xb6700d4c>'
+ addMainAgentActivationHandler : '<function addMainAgentActivationHandler at 0xb6700dbc>'
+ zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
+ __name__ : "'zc.async.agent'"
+ zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
+ __doc__ : 'None'
+ exceptions.ValueError: <zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``> not in Agent
+ *--- End of Failure #7 ---
+
+- try to make this look less frightening:
+
+ 2008-05-08T13:05:02 ERROR zc.async.events UUID ecbad1cc-1a89-11dd-8f17-0015c5e8367a already activated in queue (oid ???): another process? To stop poll attempts in this process, set ``zc.async.dispatcher.get().activated = False``. To stop polls permanently, don't start a zc.async.dispatcher!
+
+- need retry tasks, particularly for callbacks
+
+- need CRITICAL logs for callbacks
+
For future versions:
- Write the z3monitor tests.
@@ -15,6 +180,17 @@
so that the test can see what zc.async does, but zc.async can't see what
the test does. The current workaround is to start the dispatcher in the
test or the test set up (but, again, *not* The layer set up).
+ * In tests, don't check to see if poll is activated until after the first
+ poll. Try ``zc.async.testing.get_poll(zc.async.dispatcher.get(), 0)``, for
+ instance.
+ * In tests, be aware that DemoStorage does not support mvcc and does not
+ support conflict resolution, so you may experience ConflictError (write and
+ particularly read) problems with it that you will not experience as much,
+ or at all, with a storage that supports those features such as FileStorage.
+ Notice that all of the tests in this package use FileStorage.
+ * callbacks should be very, very quick, and very reliable. If you want to do
+ something that might take a while, put another job in the queue
+
For some other package, maybe:
Modified: zc.async/trunk/src/zc/async/dispatcher.py
===================================================================
--- zc.async/trunk/src/zc/async/dispatcher.py 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/src/zc/async/dispatcher.py 2008-05-09 03:33:59 UTC (rev 86555)
@@ -21,6 +21,7 @@
import twisted.python.failure
import twisted.internet.defer
import ZODB.POSException
+import ZODB.utils
import BTrees
import transaction
import transaction.interfaces
@@ -385,7 +386,7 @@
for name, agent in da.items():
job_info = []
active_jobs = [
- (job._p_oid,
+ (ZODB.utils.u64(job._p_oid),
getattr(job._p_jar.db(), 'database_name', None))
for job in agent]
agent_info = queue_info[name] = {
@@ -438,7 +439,7 @@
started_jobs.append(info)
dbname = getattr(
job._p_jar.db(), 'database_name', None)
- jobid = (job._p_oid, dbname)
+ jobid = (ZODB.utils.u64(job._p_oid), dbname)
self.jobs[jobid] = info
job_info.append(jobid)
pool.queue.put(
@@ -593,7 +594,7 @@
if database_name is None:
# these will raise ValueErrors for unknown oids. We'll let 'em.
minKey = self.jobs.minKey((oid,))
- maxKey = self.jobs.maxKey((oid,))
+ maxKey = self.jobs.maxKey((oid+1,))
if minKey != maxKey:
raise ValueError('ambiguous database name')
else:
@@ -611,13 +612,13 @@
old = []
unknown = []
for info in _iter_info(poll, queue, agent):
- res.extend(info['new jobs'])
- for job_id in info['active jobs']:
- job_info = self.jobs.get(job_id)
- if job_info is None:
- unknown.append(job_id)
- else:
- bisect.insort(old, (job_info['poll id'], job_id))
+ for jobs in (info['new jobs'], info['active jobs']):
+ for job_id in jobs:
+ job_info = self.jobs.get(job_id)
+ if job_info is None:
+ unknown.append(job_id)
+ elif not job_info['completed']:
+ bisect.insort(old, (job_info['poll id'], job_id))
res.extend(i[1] for i in old)
res.extend(unknown)
return res
Modified: zc.async/trunk/src/zc/async/dispatcher.txt
===================================================================
--- zc.async/trunk/src/zc/async/dispatcher.txt 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/src/zc/async/dispatcher.txt 2008-05-09 03:33:59 UTC (rev 86555)
@@ -243,7 +243,7 @@
{'': {'main': {'active jobs': [],
'error': None,
'len': 0,
- 'new jobs': [('\x00...', 'unnamed')],
+ 'new jobs': [(..., 'unnamed')],
'size': 3}}}
We also have some log entries.
@@ -281,7 +281,7 @@
{'':
{'main':
{'active jobs': [], 'error': None,
- 'new jobs': [('\x...', 'unnamed')], 'len': 0, 'size': 3}}}
+ 'new jobs': [(..., 'unnamed')], 'len': 0, 'size': 3}}}
[#getPollInfo]_ Notice our ``new jobs`` from the poll and the log has a value
@@ -407,11 +407,11 @@
>>> pprint.pprint(dispatcher.getStatistics()) # doctest: +ELLIPSIS
{'failed': 2,
'longest active': None,
- 'longest failed': ('\x00...', 'unnamed'),
- 'longest successful': ('\x00...', 'unnamed'),
+ 'longest failed': (..., 'unnamed'),
+ 'longest successful': (..., 'unnamed'),
'shortest active': None,
- 'shortest failed': ('\x00...', 'unnamed'),
- 'shortest successful': ('\x00...', 'unnamed'),
+ 'shortest failed': (..., 'unnamed'),
+ 'shortest successful': (..., 'unnamed'),
'started': 8,
'statistics end': datetime.datetime(...),
'statistics start': datetime.datetime(...),
Modified: zc.async/trunk/src/zc/async/monitor.py
===================================================================
--- zc.async/trunk/src/zc/async/monitor.py 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/src/zc/async/monitor.py 2008-05-09 03:33:59 UTC (rev 86555)
@@ -90,7 +90,7 @@
if uuid is not None:
uuid = uuid.UUID(uuid)
return encoder.encode(
- zc.async.dispatcher.get(uuid).getJobInfo(OID, database))
+ zc.async.dispatcher.get(uuid).getJobInfo(long(OID), database))
_find = re.compile(r'\d+[DHMS]').findall
def _dt(s):
@@ -152,7 +152,6 @@
async jobstats queue: agent:main since:1H
(results filtered to queue named '' and agent named 'main' from now
till one hour ago)"""
- # TODO: parse since and before to datetimes
if uuid is not None:
uuid = uuid.UUID(uuid)
return encoder.encode(
Modified: zc.async/trunk/src/zc/async/monitor.txt
===================================================================
--- zc.async/trunk/src/zc/async/monitor.txt 2008-05-09 03:06:10 UTC (rev 86554)
+++ zc.async/trunk/src/zc/async/monitor.txt 2008-05-09 03:33:59 UTC (rev 86555)
@@ -281,6 +281,394 @@
]
-> CLOSE
+Now that you've seen the basics, we will add some jobs and look at some of the
+statistics.
+
+ >>> def send_message():
+ ... print "imagine this sent a message to another machine"
+ >>> job = queue.put(send_message)
+ >>> import transaction
+ >>> transaction.commit()
+
+ >>> reactor.wait_for(job)
+ imagine this sent a message to another machine
+
+Here are the revised stats.
+
+ >>> connection.test_input('async jobstats\n')
+ {
+ "failed": 0,
+ "longest active": null,
+ "longest failed": null,
+ "longest successful": [
+ 30,
+ "unnamed"
+ ],
+ "shortest active": null,
+ "shortest failed": null,
+ "shortest successful": [
+ 30,
+ "unnamed"
+ ],
+ "started": 1,
+ "statistics end": "2006-08-10T15:44:22.000211Z",
+ "statistics start": "2006-08-10T15:44:27.000211Z",
+ "successful": 1,
+ "unknown": 0
+ }
+ -> CLOSE
+
+[Notice we are actually showing the oids of the jobs. For test purposes, this
+relies on implementation details of the MappingStorage, but it is currently
+practically stable, and lets us see precisely how these stats work and how to
+interpret the information]
+
+The longest successful job (the only job!) has *unpacked* oid 30, in the
+unnamed database. We can get to it using the connection and ZODB.utils.p64.
+
+ >>> import ZODB.utils
+ >>> queue._p_jar.get(ZODB.utils.p64(30)) is job
+ True
+
+So what are the details about that job? We have not used the ``job`` command
+before because we didn't have a job to look at.
+
+ >>> connection.test_input('async job 30\n') # doctest: +ELLIPSIS
+ {
+ "call": "<zc.async.job.Job (oid 30, db 'unnamed') ``zc.async.doctest_test.send_message()``>",
+ "completed": "2006-08-10T15:44:...Z",
+ "failed": false,
+ "poll id": 6420106068024891087,
+ "quota names": [],
+ "result": "None",
+ "started": "2006-08-10T15:44:...Z",
+ "thread": ...
+ }
+ -> CLOSE
+
+Here's the most recent poll.
+
+ >>> connection.test_input('async poll\n') # doctest: +ELLIPSIS
+ {
+ "key": 6420106068024891087,
+ "results": {
+ "": {
+ "main": {
+ "active jobs": [],
+ "error": null,
+ "len": 0,
+ "new jobs": [
+ [
+ 30,
+ "unnamed"
+ ]
+ ],
+ "size": 3
+ }
+ }
+ },
+ "time": "2006-08-10T15:44:...Z"
+ }
+ -> CLOSE
+
+Now let's look at the dispatcher with a job in progress.
+
+ >>> import threading
+ >>> lock = threading.Lock()
+ >>> lock.acquire()
+ True
+ >>> def wait_for_me():
+ ... lock.acquire()
+ ... print 'OK, done'
+ ...
+ >>> job = queue.put(wait_for_me)
+ >>> transaction.commit()
+ >>> reactor.wait_for(job)
+ TIME OUT
+
+ >>> _ = transaction.begin()
+ >>> job.status == zc.async.interfaces.ACTIVE
+ True
+
+Now we can use the ``jobs`` command to look at the active jobs.
+
+ >>> connection.test_input('async jobs\n')
+ [
+ [
+ 36,
+ "unnamed"
+ ]
+ ]
+ -> CLOSE
+
+Here's the information for that job.
+
+ >>> connection.test_input('async job 36\n') # doctest: +ELLIPSIS
+ {
+ "call": "<zc.async.job.Job (oid 36, db 'unnamed') ``zc.async.doctest_test.wait_for_me()``>",
+ "completed": null,
+ "failed": false,
+ "poll id": 6420106067941005007,
+ "quota names": [],
+ "result": null,
+ "started": "2006-08-10T15:44:...Z",
+ "thread": ...
+ }
+ -> CLOSE
+
+Here are the revised stats.
+
+ >>> connection.test_input('async jobstats\n') # doctest: +ELLIPSIS
+ {
+ "failed": 0,
+ "longest active": [
+ 36,
+ "unnamed"
+ ],
+ "longest failed": null,
+ "longest successful": [
+ 30,
+ "unnamed"
+ ],
+ "shortest active": [
+ 36,
+ "unnamed"
+ ],
+ "shortest failed": null,
+ "shortest successful": [
+ 30,
+ "unnamed"
+ ],
+ "started": 2,
+ "statistics end": "2006-08-10T15:44:...Z",
+ "statistics start": "2006-08-10T15:44:...Z",
+ "successful": 1,
+ "unknown": 0
+ }
+ -> CLOSE
+
+Here's our poll.
+
+ >>> connection.test_input('async poll\n') # doctest: +ELLIPSIS
+ {
+ "key": 6420106067941005007,
+ "results": {
+ "": {
+ "main": {
+ "active jobs": [],
+ "error": null,
+ "len": 0,
+ "new jobs": [
+ [
+ 36,
+ "unnamed"
+ ]
+ ],
+ "size": 3
+ }
+ }
+ },
+ "time": "2006-08-10T15:44:...Z"
+ }
+ -> CLOSE
+
+Let's wait again.
+
+ >>> reactor.wait_for(job)
+ TIME OUT
+
+ >>> _ = transaction.begin()
+ >>> job.status == zc.async.interfaces.ACTIVE
+ True
+
+Here's ``jobs`` and ``jobstats`` again.
+
+ >>> connection.test_input('async jobs\n')
+ [
+ [
+ 36,
+ "unnamed"
+ ]
+ ]
+ -> CLOSE
+
+ >>> connection.test_input('async jobstats\n') # doctest: +ELLIPSIS
+ {
+ "failed": 0,
+ "longest active": [
+ 36,
+ "unnamed"
+ ],
+ "longest failed": null,
+ "longest successful": [
+ 30,
+ "unnamed"
+ ],
+ "shortest active": [
+ 36,
+ "unnamed"
+ ],
+ "shortest failed": null,
+ "shortest successful": [
+ 30,
+ "unnamed"
+ ],
+ "started": 2,
+ "statistics end": "2006-08-10T15:44:...Z",
+ "statistics start": "2006-08-10T15:44:...Z",
+ "successful": 1,
+ "unknown": 0
+ }
+ -> CLOSE
+
+Here's the most recent poll.
+
+ >>> connection.test_input('async poll\n') # doctest: +ELLIPSIS
+ {
+ "key": 6420106067857118927,
+ "results": {
+ "": {
+ "main": {
+ "active jobs": [
+ [
+ 36,
+ "unnamed"
+ ]
+ ],
+ "error": null,
+ "len": 1,
+ "new jobs": [],
+ "size": 3
+ }
+ }
+ },
+ "time": "2006-08-10T15:44:...Z"
+ }
+ -> CLOSE
+
+We do have a few polls now. Here they are from most recent to oldest.
+
+ >>> connection.test_input('async polls\n') # doctest: +ELLIPSIS
+ [
+ {
+ "key": 6420106067857118927,
+ "results": {
+ "": {
+ "main": {
+ "active jobs": [
+ [
+ 36,
+ "unnamed"
+ ]
+ ],
+ "error": null,
+ "len": 1,
+ "new jobs": [],
+ "size": 3
+ }
+ }
+ },
+ "time": "2006-08-10T15:44:...Z"
+ },
+ {
+ "key": 6420106067941005007,
+ "results": {
+ "": {
+ "main": {
+ "active jobs": [],
+ "error": null,
+ "len": 0,
+ "new jobs": [
+ [
+ 36,
+ "unnamed"
+ ]
+ ],
+ "size": 3
+ }
+ }
+ },
+ "time": "2006-08-10T15:44:...Z"
+ },
+ {
+ "key": 6420106068024891087,
+ "results": {
+ "": {
+ "main": {
+ "active jobs": [],
+ "error": null,
+ "len": 0,
+ "new jobs": [
+ [
+ 30,
+ "unnamed"
+ ]
+ ],
+ "size": 3
+ }
+ }
+ },
+ "time": "2006-08-10T15:44:...Z"
+ }
+ ]
+ -> CLOSE
+
+Now we'll let the job complete.
+
+ >>> lock.release()
+ >>> reactor.wait_for(job)
+ OK, done
+
+The job is done according to the ``job`` command also.
+
+ >>> connection.test_input('async job 36\n') # doctest: +ELLIPSIS
+ {
+ "call": "<zc.async.job.Job (oid 36, db 'unnamed') ``zc.async.doctest_test.wait_for_me()``>",
+ "completed": "2006-08-10T15:44:...Z",
+ "failed": false,
+ "poll id": 6420106067941005007,
+ "quota names": [],
+ "result": "None",
+ "started": "2006-08-10T15:44:...Z",
+ "thread": ...
+ }
+ -> CLOSE
+
+No more active jobs.
+
+ >>> connection.test_input('async jobs\n')
+ []
+ -> CLOSE
+
+Here are the revised stats.
+
+ >>> connection.test_input('async jobstats\n') # doctest: +ELLIPSIS
+ {
+ "failed": 0,
+ "longest active": null,
+ "longest failed": null,
+ "longest successful": [
+ 36,
+ "unnamed"
+ ],
+ "shortest active": null,
+ "shortest failed": null,
+ "shortest successful": [
+ 30,
+ "unnamed"
+ ],
+ "started": 2,
+ "statistics end": "2006-08-10T15:44:...Z",
+ "statistics start": "2006-08-10T15:44:...Z",
+ "successful": 2,
+ "unknown": 0
+ }
+ -> CLOSE
+
+ >>> reactor.stop()
+ >>> import time
+ >>> time.sleep(1)
+
.. [#setUp] See the discussion in other documentation to explain this code.
>>> import ZODB.FileStorage
More information about the Checkins
mailing list