[Checkins] SVN: zc.async/trunk/s put a "tips and tricks" section in
main page. Prepare for 1.3.0.
Gary Poster
gary at zope.com
Fri Jul 4 21:26:03 EDT 2008
Log message for revision 88030:
put a "tips and tricks" section in main page. Prepare for 1.3.0.
Changed:
U zc.async/trunk/setup.py
U zc.async/trunk/src/zc/async/CHANGES.txt
U zc.async/trunk/src/zc/async/README.txt
U zc.async/trunk/src/zc/async/README_2.txt
U zc.async/trunk/src/zc/async/README_3.txt
U zc.async/trunk/src/zc/async/catastrophes.txt
U zc.async/trunk/src/zc/async/ftesting.txt
U zc.async/trunk/src/zc/async/tips.txt
U zc.async/trunk/src/zc/async/z3.txt
-=-
Modified: zc.async/trunk/setup.py
===================================================================
--- zc.async/trunk/setup.py 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/setup.py 2008-07-05 01:26:03 UTC (rev 88030)
@@ -71,7 +71,7 @@
setup(
name='zc.async',
- version='1.3a7',
+ version='1.3.0',
packages=find_packages('src'),
package_dir={'':'src'},
zip_safe=False,
@@ -83,6 +83,10 @@
'src/zc/async/README_2.txt',
'src/zc/async/README_3.txt',
'src/zc/async/README_3b.txt',
+ 'src/zc/async/tips.txt',
+ 'src/zc/async/catastrophes.txt',
+ 'src/zc/async/z3.txt',
+ 'src/zc/async/ftesting.txt',
"=======\nChanges\n=======\n\n",
'src/zc/async/CHANGES.txt',
out=True),
Modified: zc.async/trunk/src/zc/async/CHANGES.txt
===================================================================
--- zc.async/trunk/src/zc/async/CHANGES.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/CHANGES.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -1,6 +1,8 @@
-1.3 (unreleased)
+1.3 (2008-07-04)
================
+- added "Tips and Tricks" and incorporated into the PyPI page.
+
- added ``setUp`` and ``tearDown`` hooks to Job class so that code can run
before and after the main job's code. The output of ``setUp`` is passed as
an argument to ``tearDown`` so that one can pass state to the other, if
Modified: zc.async/trunk/src/zc/async/README.txt
===================================================================
--- zc.async/trunk/src/zc/async/README.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/README.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -3,6 +3,7 @@
~~~~~~~~
.. contents::
+ :depth: 2
============
Introduction
@@ -127,8 +128,8 @@
Reading More
============
-This document continues on with three other main sections: `Usage`_,
-`Configuration without Zope 3`_, and `Configuration with Zope 3`_.
+This document continues on with four other main sections: `Usage`_,
+`Configuration`_, `Configuration with Zope 3`_, and `Tips and Tricks`_.
Other documents in the package are primarily geared as maintainer
documentation, though the author has tried to make them readable and
@@ -867,7 +868,7 @@
This concludes our discussion of zc.async usage. The `next section`_ shows how
to configure zc.async without Zope 3 [#stop_usage_reactor]_.
-.. _next section: `Configuration without Zope 3`_
+.. _next section: `Configuration`_
.. ......... ..
.. Footnotes ..
Modified: zc.async/trunk/src/zc/async/README_2.txt
===================================================================
--- zc.async/trunk/src/zc/async/README_2.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/README_2.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -1,6 +1,6 @@
-============================
-Configuration without Zope 3
-============================
+=============
+Configuration
+=============
This section discusses setting up zc.async without Zope 3. Since Zope 3 is
ill-defined, we will be more specific: this describes setting up zc.async
@@ -43,7 +43,7 @@
The required registrations can be installed for you by the
``zc.async.configure.base`` function. Most other examples in this package,
such as those in the `Usage`_ section, use this in their
-test setup.
+test setup.
Again, for a quick start, you might just want to use the helper
``zc.async.configure.base`` function, and move on to the `Required ZODB Set
@@ -58,7 +58,7 @@
The ``zc.twist`` package provides all of these adapters. However,
zope.app.keyreference also provides a version of the ``connection`` adapter
-that is identical or very similar, and that should work fine if you are
+that is identical or very similar, and that should work fine if you are
already using that package in your application.
>>> import zc.twist
@@ -92,7 +92,7 @@
and may want to use it to optionally filter jobs during ``claim`` in the
future. Also, the dispatcher will look for a UUID utility if a UUID is not
specifically provided to its constructor.
-
+
>>> from zc.async.instanceuuid import UUID
>>> zope.component.provideUtility(
... UUID, zc.async.interfaces.IUUID, '')
@@ -133,7 +133,7 @@
A more subtle decision is whether to use multiple databases. The zc.async
dispatcher can generate a lot of database churn. It may be wise to put the
-queue in a separate database from your content database(s).
+queue in a separate database from your content database(s).
The downsides to this option include the fact that you must be careful to
specify to which database objects belong; and that broken cross-database
@@ -153,11 +153,11 @@
>>> import ZODB.FileStorage
>>> storage = ZODB.FileStorage.FileStorage(
... 'main.fs', create=True)
-
+
>>> async_storage = ZODB.FileStorage.FileStorage(
... 'async.fs', create=True)
- >>> from ZODB.DB import DB
+ >>> from ZODB.DB import DB
>>> databases[''] = db = DB(storage)
>>> databases['async'] = async_db = DB(async_storage)
>>> async_db.databases = db.databases = databases
@@ -181,7 +181,7 @@
see that in use in `Configuration with Zope 3`_. For now, though, we're taking
things step by step and explaining what's going on.
-Dispatchers look for queues in a mapping off the root of the database in
+Dispatchers look for queues in a mapping off the root of the database in
a key defined as a constant: zc.async.interfaces.KEY. This mapping should
generally be a zc.async.queue.Queues object.
@@ -525,62 +525,62 @@
- pytz
A Python time zone library
-
+
- rwproperty
A small package of descriptor conveniences
-
+
- uuid
The uuid module included in Python 2.5
-
+
- zc.dict
A ZODB-aware dict implementation based on BTrees.
-
+
- zc.queue
A ZODB-aware queue
-
+
- zc.twist
Conveniences for working with Twisted and the ZODB
-
+
- twisted
The Twisted internet library.
-
+
- ZConfig
A general configuration package coming from the Zope project with which
the ZODB tests.
-
+
- zdaemon
A general daemon tool coming from the Zope project.
-
+
- ZODB3
The Zope Object Database.
-
+
- zope.bforest
Aggregations of multiple BTrees into a single dict-like structure,
reasonable for rotating data structures, among other purposes.
-
+
- zope.component
A way to hook together code by contract.
-
+
- zope.deferredimport
A way to defer imports in Python packages, often to prevent circular
import problems.
-
+
- zope.deprecation
A small framework for deprecating features.
-
+
- zope.event
An exceedingly small event framework that derives its power from
zope.component.
-
+
- zope.i18nmessageid
A way to specify strings to be translated.
-
+
- zope.interface
A way to specify code contracts and other data structures.
-
+
- zope.proxy
A way to proxy other Python objects.
-
+
- zope.testing
Testing extensions and helpers.
@@ -722,7 +722,7 @@
>>> reactor.callFromThread(reactor.stop)
>>> thread.join(3)
>>> assert not dispatcher.activated, 'dispatcher did not deactivate'
-
+
>>> import zc.async.testing
>>> reactor = zc.async.testing.Reactor()
>>> dispatcher.reactor = reactor
Modified: zc.async/trunk/src/zc/async/README_3.txt
===================================================================
--- zc.async/trunk/src/zc/async/README_3.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/README_3.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -63,31 +63,31 @@
... path %(main_storage_path)s
... </filestorage>
... </zodb>
- ...
+ ...
... <product-config zc.z3monitor>
... port %(monitor_port)s
... </product-config>
- ...
+ ...
... <logger>
... level debug
... name zc.async
... propagate no
- ...
+ ...
... <logfile>
... path %(async_event_log)s
... </logfile>
... </logger>
- ...
+ ...
... <logger>
... level debug
... name zc.async.trace
... propagate no
- ...
+ ...
... <logfile>
... path %(async_trace_log)s
... </logfile>
... </logger>
- ...
+ ...
... <eventlog>
... <logfile>
... formatter zope.exceptions.log.Formatter
@@ -105,7 +105,7 @@
... 'event_log': os.path.join(dir, 'z3.log'),
... 'async_event_log': os.path.join(dir, 'async.log'),
... 'async_trace_log': os.path.join(dir, 'async_trace.log'),}
- ...
+ ...
In a non-trivial production system, you will also probably want to replace
the file storage with a <zeoclient> stanza.
@@ -180,18 +180,18 @@
>>> import telnetlib
>>> tn = telnetlib.Telnet('127.0.0.1', monitor_port)
>>> tn.write('async status\n') # immediately disconnects
- >>> print tn.read_all() # doctest: +ELLIPSIS
+ >>> print tn.read_all() # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
{
"poll interval": {
"seconds": ...
- },
- "status": "RUNNING",
+ },
+ "status": "RUNNING",
"time since last poll": {
"seconds": ...
- },
+ },
"uptime": {
"seconds": ...
- },
+ },
"uuid": "..."
}
<BLANKLINE>
@@ -238,10 +238,10 @@
.. [#extras_require] The "[z3]" is an "extra", defined in zc.async's setup.py
in ``extras_require``. It pulls along zc.z3monitor and simplejson in
- addition to the packages described in the `Configuration without Zope 3`_
- section. Unfortunately, zc.z3monitor depends on zope.app.appsetup, which as
- of this writing ends up depending indirectly on many, many packages, some
- as far flung as zope.app.rotterdam.
+ addition to the packages described in the `Configuration`_ section.
+ Unfortunately, zc.z3monitor depends on zope.app.appsetup, which as of this
+ writing ends up depending indirectly on many, many packages, some as far
+ flung as zope.app.rotterdam.
.. [#get_vals]
Modified: zc.async/trunk/src/zc/async/catastrophes.txt
===================================================================
--- zc.async/trunk/src/zc/async/catastrophes.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/catastrophes.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -1,9 +1,9 @@
Recovering from Catastrophes
============================
-------------
-Introduction
-------------
+--------------------
+What Might Go Wrong?
+--------------------
Sometimes bad things happen in the course of processing tasks. What might go
wrong? How does zc.async handle these errors? What are your responsibilities?
@@ -46,8 +46,8 @@
What needs to happen to handle these problems?
-Process Ends
-............
+Process Ends while Polling
+..........................
If the process ends, your daemonizing front-end (zdaemon, supervisor, etc.)
needs to restart it. The ZODB will discard incomplete transaction data, if any.
@@ -64,8 +64,8 @@
restarts a process for the original dispatcher instance, the new process
needs to realize that the old process is dead, not competing with it.
-Transaction Error
-.................
+Transaction Error while Polling
+...............................
If the poll gets a conflict error, it should simply abort and retry the poll,
forever, with a small back-off.
@@ -139,8 +139,8 @@
react to the problem in a callback, and analyze it later. This is discussed in
detail in other documents.
-Process Ends
-............
+Process Ends During Job
+.......................
If a process ends while it is performing a job, that is similar, in large part,
to the possibility of the process ending the polling job: we need to restart
@@ -168,8 +168,8 @@
These settings are controlled with a RetryPolicy, discussed below.
-Transaction Error
-.................
+Transaction Error During Job
+............................
Handling transaction errors after processing a job is also similar to the
handling of transaction errors for polling exceptions. ConflictErrors and
@@ -264,20 +264,23 @@
zc.async currently ships with three retry policies.
1. The default, appropriate for most fully transactional jobs, is the
- zc.async.job.RetryCommonFourTimes.
+ zc.async.job.RetryCommonFourTimes. This retries ZEO disconnects forever;
+ and interrupts and transaction errors such as conflicts a set number of
+ times.
2. The other available (pre-written) option for transactional jobs is
zc.async.job.RetryCommonForever. Callbacks will get this policy by
- default.
+ default. This retries ZEO disconnects, transaction errors such as conflict
+ errors, interrupts, and *anything* that happens during the job's commit,
+ forever.
-Both of these policies retry ZEO disconnects forever; and interrupts and
-transaction errors such as conflicts either four times (for a total of five
-attempts) or forever, respectively.
-
3. The last retry policy is zc.async.job.NeverRetry. This is appropriate for
non-transactional jobs. You'll still typically need to handle errors in
your callbacks.
+If you look at these, you will see that it is trivial to write your own, if
+desired.
+
Scenarios
---------
Modified: zc.async/trunk/src/zc/async/ftesting.txt
===================================================================
--- zc.async/trunk/src/zc/async/ftesting.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/ftesting.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -1,3 +1,6 @@
+Zope 3 Testing Tips and Tricks
+==============================
+
Normally, in a Zope 3 configuration that uses zc.async, you configure it
when you start your application. For instance, you might include a zc.async
zcml file like basic_dispatcher_policy.zcml that performs the necessary set up.
Modified: zc.async/trunk/src/zc/async/tips.txt
===================================================================
--- zc.async/trunk/src/zc/async/tips.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/tips.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -2,20 +2,24 @@
Tips and Tricks
===============
-General
-=======
+General Tips and Tricks
+=======================
-* avoid long transactions if possible. really avoid long transactions
- involving frequently written objects. Discuss ramifications and
- strategies, such as doing big work in one job, then in callback schedule
- actually writing the data into the hotspot. XXX
+* Avoid long transactions if possible. Really try to avoid long transactions
+ involving frequently written objects. One possible strategy is to divide up
+ your code into a job for low-conflict tasks and one or more jobs for
+ high-conflict tasks, perhaps created in a callback.
-* callbacks should be very, very quick, and very reliable. If you want to do
- something that might take a while, put another job in the queue
+* Callbacks should be quick and reliable. If you want to do something that
+ might take a while, put another job in the queue.
-Testing
-=======
+* Some tasks are non-transactional. If you want to do them in a ``Job``, you
+ don't want them to be retried! Use the NeverRetry retry policy for these,
+ as described in the `Recovering from Catastrophes`_ section below.
+Testing Tips and Tricks
+=======================
+
* In tests, don't check to see if poll is activated until after the first
poll. Try ``zc.async.testing.get_poll(zc.async.dispatcher.get(), 0)``, for
instance.
@@ -26,6 +30,23 @@
or at all, with a storage that supports those features such as FileStorage.
Notice that all of the tests in this package use FileStorage.
+* ``zc.async.dispatcher.get()`` will get you the dispatcher. You can then check
+ if it is ``activated`` and also use the other introspection and status
+ methods.
+
+* The ``zc.async.testing`` module has a number of helpful functions for
+ testing. ``get_poll``, given a dispatcher, will give you the next poll. This
+ is a good way to make sure that a job you just put in has had a chance to be
+ claimed by a dispatcher. It's also a reasonable way to verify that the
+ dispatcher has started. Other useful testing functions are
+ ``zc.async.testing.wait_for_result``, which waits for the result on a give
+ job and returns it; and ``zc.async.testing.wait_for_annotation``, which waits
+ for a given annotation on a given job. These are demonstrated in various
+ doctests in this package, but should also be reasonably simple and
+ self-explanatory.
+
+.. insert the catastrophes.txt document here
+
.. insert the z3.txt document here
.. insert the ftesting.txt document here.
Modified: zc.async/trunk/src/zc/async/z3.txt
===================================================================
--- zc.async/trunk/src/zc/async/z3.txt 2008-07-05 00:59:12 UTC (rev 88029)
+++ zc.async/trunk/src/zc/async/z3.txt 2008-07-05 01:26:03 UTC (rev 88030)
@@ -1,6 +1,5 @@
-----------------------------
-The Context-Aware Zope 3 Job
-----------------------------
+Zope 3 General Tips and Tricks
+==============================
If you use Zope 3, sometimes you want async jobs that have local sites and
security set up. ``zc.async.z3.Job`` is a subclass of the main
More information about the Checkins
mailing list