[Checkins] SVN: zc.async/branches/dev/src/zc/async/ rip out a bunch
of repeated setup and put it in a configure.py;
edit README_2 heavily
Gary Poster
gary at zope.com
Sun Apr 6 22:16:30 EDT 2008
Log message for revision 85130:
rip out a bunch of repeated setup and put it in a configure.py; edit README_2 heavily
Changed:
U zc.async/branches/dev/src/zc/async/README.txt
U zc.async/branches/dev/src/zc/async/README_2.txt
U zc.async/branches/dev/src/zc/async/agent.txt
A zc.async/branches/dev/src/zc/async/configure.py
U zc.async/branches/dev/src/zc/async/dispatcher.txt
U zc.async/branches/dev/src/zc/async/job.txt
U zc.async/branches/dev/src/zc/async/jobs_and_transactions.txt
U zc.async/branches/dev/src/zc/async/monitor.txt
U zc.async/branches/dev/src/zc/async/queue.txt
U zc.async/branches/dev/src/zc/async/subscribers.txt
-=-
Modified: zc.async/branches/dev/src/zc/async/README.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/README.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/README.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -933,32 +933,24 @@
register IPersistent to ITransactionManager because the adapter is
designed for it.
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
+ We also need to be able to get data manager partials for functions and
+ methods; normal partials for functions and methods; and a data manager for
+ a partial. Here are the necessary registrations.
- We need to be able to get data manager partials for functions and methods;
- normal partials for functions and methods; and a data manager for a partial.
- Here are the necessary registrations.
+ The dispatcher will look for a UUID utility, so we also need one of these.
+
+ The ``zc.async.configure.base`` function performs all of these
+ registrations. If you are working with zc.async without ZCML you might want
+ to use it or ``zc.async.configure.minimal`` as a convenience.
- >>> import zope.component
- >>> import types
- >>> import zc.async.interfaces
- >>> import zc.async.job
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
- ...
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
+ Now we'll set up the database, and make some policy decisions. As
+ the subsequent ``configuration`` sections discuss, some helpers are
+ available for you to set this stuff up if you'd like, though it's not too
+ onerous to do it by hand.
+
We'll use a test reactor that we can control.
>>> import zc.async.testing
@@ -987,24 +979,23 @@
>>> import transaction
>>> transaction.commit()
- The dispatcher will look for a UUID utility.
-
- >>> from zc.async.instanceuuid import UUID
- >>> import zope.component
- >>> zope.component.provideUtility(
- ... UUID, zc.async.interfaces.IUUID, '')
-
Now we can instantiate, activate, and perform some reactor work in order
to let the dispatcher register with the queue.
>>> import zc.async.dispatcher
>>> dispatcher = zc.async.dispatcher.Dispatcher(db, reactor)
- >>> dispatcher.UUID == UUID
- True
>>> dispatcher.activate()
>>> reactor.time_flies(1)
1
+ The UUID is set on the dispatcher.
+
+ >>> import zope.component
+ >>> import zc.async.interfaces
+ >>> UUID = zope.component.getUtility(zc.async.interfaces.IUUID)
+ >>> dispatcher.UUID == UUID
+ True
+
Here's an agent named 'main'
>>> import zc.async.agent
Modified: zc.async/branches/dev/src/zc/async/README_2.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/README_2.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/README_2.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -2,30 +2,61 @@
Configuration Without Zope 3
============================
-This section discusses setting up zc.async without Zope 3. Since Zope 3
-is ill-defined, we will be more specific: this describes setting up
-zc.async without ZCML, without any zope.app packages, and with as few
-dependencies as possible. A casual way of describing the dependencies
-is "ZODB and zope.component"[#specific_dependencies]_.
+This section discusses setting up zc.async without Zope 3. Since Zope 3 is
+ill-defined, we will be more specific: this describes setting up zc.async
+without ZCML, without any zope.app packages, and with as few dependencies as
+possible. A casual way of describing the dependencies is "ZODB and
+zope.component"[#specific_dependencies]_.
The next section, `Configuration With Zope 3`_, still tries to limit
-dependencies, but includes both ZCML and indirect and direct
-dependencies on such packages as zope.publisher and zope.app.appsetup.
+dependencies, but includes both ZCML and indirect and direct dependencies on a
+few "zope.app.*" packages like zope.app.appsetup. It still is minimal enough
+that someone wanting to avoid ZCML, for instance, might still find valuable
+information.
-Configuration has three basic parts: component registrations, ZODB
-setup, and ZODB configuration.
+You may have one or two kinds of configurations for your software using
+zc.async. The simplest approach is to have all processes able both to put items
+in queues, and to perform them with a dispatcher. You can then use on-the-fly
+ZODB configuration to determine what jobs, if any, each process' dispatcher
+performs. If a dispatcher has no agents in a given queue, as we'll discuss
+below, the dispatcher will not perform any job for that queue.
-Component Registrations
-=======================
+However, if you want to create some processes that can only put items in a
+queue, and do not have a dispatcher at all, that is easy to do. We'll call this
+a "client" process, and the full configuration a "client/server process". As
+you might expect, the configuration of a client process is a subset of the
+configuration of the client/server process.
-Some registrations are required, and some are optional. Since they are
-component registrations, even for the required registrations, other
-implementations are possible.
+We will first describe setting up a client, non-dispatcher process, in which
+you only can put items in a zc.async queue; and then describe setting up a
+dispatcher client/server process that can be used both to request and to
+perform jobs.
---------
-Required
---------
+Configuring a Client Process
+============================
+Generally, zc.async configuration has four basic parts: component
+registrations, ZODB setup, ZODB configuration, and process configuration. For
+a client process, we'll discuss required component registrations; ZODB
+setup; minimal ZODB configuration; process configuration; and then circle
+back around for some optional component registrations.
+
+--------------------------------
+Required Component Registrations
+--------------------------------
+
+The required registrations can be installed for you by the
+``zc.async.configure.base`` function. Most other documents in this package,
+such as those in the "Usage" section (found in README.txt), use this in their
+test setup.
+
+**Again, for a quick start, you might just want to use the helper
+``zc.async.configure.base`` function, and move on to the ``Required ZODB Set
+Up``_ section below.**
+
+Here, though, we will go over each required registration to briefly explain
+what they are.
+
You must have three adapter registrations: IConnection to
ITransactionManager, IPersistent to IConnection, and IPersistent to
ITransactionManager.
@@ -35,13 +66,13 @@
that is identical or very similar, and that should work fine if you are
already using that package in your application.
- >>> from zc.twist import transactionManager, connection
+ >>> import zc.twist
>>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
+ >>> zope.component.provideAdapter(zc.twist.transactionManager)
+ >>> zope.component.provideAdapter(zc.twist.connection)
>>> import ZODB.interfaces
>>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
+ ... zc.twist.transactionManager, adapts=(ZODB.interfaces.IConnection,))
We also need to be able to adapt functions and methods to jobs. The
zc.async.job.Job class is the expected implementation.
@@ -57,17 +88,15 @@
... zc.async.job.Job,
... adapts=(types.MethodType,),
... provides=zc.async.interfaces.IJob)
- ...
+ >>> zope.component.provideAdapter( # optional, rarely used
+ ... zc.async.job.Job,
+ ... adapts=(zc.twist.METHOD_WRAPPER_TYPE,),
+ ... provides=zc.async.interfaces.IJob)
---------
-Optional
---------
-
-UUID
-----
-
-The dispatcher will look for a UUID utility if a UUID is not specifically
-provided to its constructor.
+The queue looks for the UUID utility to set the ``assignerUUID`` job attribute,
+and may want to use it to optionally filter jobs during ``claim`` in the
+future. Also, the dispatcher will look for a UUID utility if a UUID is not
+specifically provided to its constructor.
>>> from zc.async.instanceuuid import UUID
>>> zope.component.provideUtility(
@@ -77,7 +106,7 @@
to uniquely identify the process when in production. It is stored in
the file specified by the ZC_ASYNC_UUID environment variable (or in
``os.join(os.getcwd(), 'uuid.txt')`` if this is not specified, for easy
-experimentation.
+experimentation).
>>> import uuid
>>> import os
@@ -88,69 +117,25 @@
>>> UUID == uuid
True
-The uuid.txt file is intended to stay in the instance home as a
-persistent identifier.
+The uuid.txt file is intended to stay in the instance home as a persistent
+identifier.
-Queue Adapter
--------------
+Again, all of the required registrations above can be accomplished quickly with
+``zc.async.configure.base``.
-You may want to set up an adapter from persistent objects to a named queue.
-The zc.async.queue.getDefaultQueue adapter is a reasonable approach.
-
- >>> import zc.async.queue
- >>> zope.component.provideAdapter(zc.async.queue.getDefaultQueue)
-
-This returns the queue names '' (empty string).
-
-Agent Subscribers
------------------
-
-As we'll see below, the dispatcher fires an event when it registers with
-a queue, and another when it activates the queue. These events give you
-the opportunity to register subscribers to add one or more agents to a
-queue, to tell the dispatcher what jobs to perform.
-zc.async.agent.addMainAgentActivationHandler is a reasonable starter: it
-adds a single agent named 'main' if one does not exist. The agent has a
-simple indiscriminate FIFO policy for the queue. If you want to write
-your own subscriber, look at this.
-
-Agents are an important part of the ZODB configuration, and so are described
-more in depth below.
-
- >>> import zc.async.agent
- >>> zope.component.provideHandler(
- ... zc.async.agent.addMainAgentActivationHandler)
-
-This subscriber is registered for the IDispatcherActivated event; another
-approach might use the IDispatcherRegistered event.
-
-Database Startup Subscribers
-----------------------------
-
-Typically you will want to start the reactor, if necessary, and instantiate
-and activate the dispatcher when the database is ready. Depending on your
-application, this can be done in-line with your start up code, or with a
-subscriber to some event.
-
-Zope 3 provides an event, zope.app.appsetup.interfaces.IDatabaseOpenedEvent,
-that the Zope 3 configuration uses. You may also want to follow this kind
-of pattern.
-
-For our example, we will start the dispatcher in-line (see the beginning of
-the `ZODB Configuration`_ section).
-
-ZODB Setup
-==========
-
--------------------
-Storage and DB Setup
+Required ZODB Set Up
--------------------
On a basic level, zc.async needs a setup that supports good conflict
resolution. Most or all production ZODB storages now have the necessary
-APIs to support MVCC. You should also make sure that your ZEO server
-has all the code that includes conflict resolution, such as zc.queue.
+APIs to support MVCC.
+Of course, if you want to run multiple processes, you need ZEO. You should also
+then make sure that your ZEO server installation has all the code that includes
+conflict resolution, such as zc.queue, because, as of this writing, conflict
+resolution happens in the ZEO server, not in clients.
+
A more subtle decision is whether to use multiple databases. The zc.async
dispatcher can generate a lot of database churn. It may be wise to put the
queue in a separate database from your content database(s).
@@ -159,12 +144,15 @@
specify to which database objects belong; and that broken cross-database
references are not handled gracefully in the ZODB as of this writing.
-We will use multiple databases for our example here. See the footnote in
-the usage section that sets up the tests for a non-multiple database
-approach.
+We will use multiple databases for our example here, because we are trying to
+demonstrate production-quality examples. We will show this with a pure-Python
+approach, rather than the ZConfig approach usually used by Zope. If you know
+ZConfig, that will be a reasonable approach as well; see zope.app.appsetup
+for how Zope uses ZConfig to set up multidatabases.
-(We use a FileStorage rather than a MappingStorage variant typical in
-tests and examples because we want MVCC, as mentioned above.)
+In our example, we create two file storages. In production, you might likely
+use ZEO; hooking ClientStorage up instead of FileStorage should be straight
+forward.
>>> databases = {}
>>> import ZODB.FileStorage
@@ -183,10 +171,21 @@
>>> conn = db.open()
>>> root = conn.root()
----------
-DB layout
----------
+------------------
+ZODB Configuration
+------------------
+A Queue
+-------
+
+All we must have for a client to be able to put jobs in a queue is...a queue.
+
+For a quick start, the ``zc.async.subscribers`` module provides some a subscriber to
+a DatabaseOpened event that does the right dance. See
+``multidb_queue_installer`` and ``queue_installer`` in that module, and you can
+see that in use in the Zope 3 configuration section (in README_3). For now,
+though, we're taking things step by step and explaining what's going on.
+
Dispatchers look for queues in a mapping off the root of the database in
a key defined as a constant: zc.async.interfaces.KEY. This mapping should
generally be a zc.async.queue.Queues object.
@@ -199,6 +198,7 @@
key.
>>> conn2 = conn.get_connection('async')
+ >>> import zc.async.queue
>>> queues = conn2.root()['mounted_queues'] = zc.async.queue.Queues()
Note that the 'mounted_queues' key in the async database is arbitrary:
@@ -221,19 +221,148 @@
>>> queue = queues[''] = zc.async.queue.Queue()
>>> transaction.commit()
-We can now get the queue with the optional adapter from IPersistent to IQueue
-above.
+Quotas
+------
- >>> queue is zc.async.interfaces.IQueue(root)
+We touched on quotas in the usage section. Some jobs will need to
+access resoources that are shared across processes. A central data
+structure such as an index in the ZODB is a prime example, but other
+examples might include a network service that only allows a certain
+number of concurrent connections. These scenarios can be helped by
+quotas.
+
+Quotas are demonstrated in the usage section. For configuration, you
+should know these characteristics:
+
+- you cannot add a job with a quota name that is not defined in the
+ queue[#undefined_quota_name]_;
+
+- you cannot add a quota name to a job in a queue if the quota name is not
+ defined in the queue[#no_mutation_to_undefined]_;
+
+- you can create and remove quotas on the queue[#create_remove_quotas]_;
+
+- you can remove quotas if pending jobs have their quota names--the quota name
+ is then ignored[#remove_quotas]_;
+
+- quotas default to a size of 1[#default_size]_;
+
+- this can be changed at creation or later[#change_size]_; and
+
+- decreasing the size of a quota while the old quota size is filled will
+ not affect the currently running jobs[#decreasing_affects_future]_.
+
+Multiple Queues
+---------------
+
+Since we put our queues in a mapping of them, we can also create multiple
+queues. This can make some scenarios more convenient and simpler to reason
+about. For instance, while you might have agents filtering jobs as we
+describe above, it might be simpler to say that you have a queue for one kind
+of job--say, processing a video file or an audio file--and a queue for other
+kinds of jobs. Then it is easy and obvious to set up simple FIFO agents
+as desired for different dispatchers. The same kind of logic could be
+accomplished with agents, but it is easier to picture the multiple queues.
+
+Another use case for multiple queues might be for specialized queues, like ones
+that broadcast jobs. You could write a queue subclass that broadcasts copies of
+jobs they get to all dispatchers, aggregating results. This could be used to
+send "events" to all processes, or to gather statistics on certain processes,
+and so on.
+
+Generally, any time the application wants to be able to assert a kind of job
+rather than letting the agents decide what to do, having separate queues is
+a reasonable tool.
+
+---------------------
+Process Configuration
+---------------------
+
+Daemonization
+-------------
+
+You often want to daemonize your software, so that you can restart it if
+there's a problem, keep track of it and monitor it, and so on. ZDaemon
+(http://pypi.python.org/pypi/zdaemon) and Supervisor (http://supervisord.org/)
+are two fairly simple ways of doing this for both client and client/server
+processes. If your main application can be packaged as a setuptools
+distribution (egg or source release or even development egg) then you can
+have your main application as a zc.async client and your dispatchers running
+a separate zc.async-only main loop that simply includes your main application
+as a dependency, so the necessary software is around. You may have to do a
+bit more configuration on the client/server side to mimic global registries
+such as zope.component registrations and so on between the client and the
+client/servers, but this shouldn't be too bad.
+
+UUID File Location
+------------------
+
+As discussed above, the instanceuuid module will look for an environmental
+variable ``ZC_ASYNC_UUID`` to find the file name to use, and failing that will
+use ``os.join(os.getcwd(), 'uuid.txt')``. It's worth noting that daemonization
+tools such as ZDaemon and Supervisor (3 or greater) make setting environment
+values for child processes an easy (and repeatable) configuration file setting.
+
+-----------------------------------------------------
+Optional Component Registrations for a Client Process
+-----------------------------------------------------
+
+The only optional component registration potentially valuable for client
+instances that only put jobs in the queue is registering an adapter from
+persistent objects to a queue. The ``zc.async.queue.getDefaultQueue`` adapter
+does this for an adapter to the queue named '' (empty string). Since that's
+what we have from the `ZODB Configuration`_ above section, we'll register it.
+Writing your own adapter is trivial, as you can see if you look at the
+implementation of this function.
+
+ >>> zope.component.provideAdapter(zc.async.queue.getDefaultQueue)
+ >>> zc.async.interfaces.IQueue(root) is queue
True
-ZODB Configuration
-==================
+Configuring a Client/Server Process
+===================================
-Now we can start the reactor, and start the dispatcher. As noted above,
-in some applications this may be done with an event subscriber. We will
-do it inline.
+Configuring a client/server process--something that includes a running
+dispatcher--means doing everything described above, plus a bit more. You
+need to set up and start a reactor and dispatcher; configure agents as desired
+to get the dispatcher to do some work; and optionally configure logging.
+For a quick start, the ``zc.async.subscribers`` module have some conveniences
+to start a threaded reactor and dispatcher, and to install agents. You might
+want to look at those to get started. They are also used in the Zope 3
+configuration (README_3). Meanwhile, this document continues to go
+step-by-step instead, to try and explain the components and configuration.
+
+Even though it seems reasonable to first start a dispatcher and then set up its
+agents, we'll first define a subscriber to create an agent. As we'll see below,
+the dispatcher fires an event when it registers with a queue, and another when
+it activates the queue. These events give you the opportunity to register
+subscribers to add one or more agents to a queue, to tell the dispatcher what
+jobs to perform. zc.async.agent.addMainAgentActivationHandler is a reasonable
+starter: it adds a single agent named 'main' if one does not exist. The agent
+has a simple indiscriminate FIFO policy for the queue. If you want to write
+your own subscriber, look at this, or at the more generic subscriber in the
+``zc.async.subscribers`` module.
+
+Agents are an important part of the ZODB configuration, and so are described
+more in depth below.
+
+ >>> import zc.async.agent
+ >>> zope.component.provideHandler(
+ ... zc.async.agent.addMainAgentActivationHandler)
+
+This subscriber is registered for the IDispatcherActivated event; another
+approach might use the IDispatcherRegistered event.
+
+-----------------------
+Starting the Dispatcher
+-----------------------
+
+Now we can start the reactor, and start the dispatcher.
+In some applications this may be done with an event subscriber to
+DatabaseOpened, as is done in ``zc.async.subscribers``. Here, we will do it
+inline.
+
Any object that conforms to the specification of zc.async.interfaces.IReactor
will be usable by the dispatcher. For our example, we will use our own instance
of the Twisted select-based reactor running in a separate thread. This is
@@ -241,14 +370,15 @@
so this approach can be used with an application that does not otherwise use
Twisted (for instance, a Zope application using the "classic" zope publisher).
-The testing module also has a reactor on which the `Usage` section relies.
+The testing module also has a reactor on which the `Usage` section relies, if
+you would like to see a minimal contract.
Configuring the basics is fairly simple, as we'll see in a moment. The
trickiest part is to handle signals cleanly. Here we install signal
handlers in the main thread using ``reactor._handleSignals``. This may
work in some real-world applications, but if your application already
-needs to handle signals you may need a more careful approach. The Zope
-3 configuration has some options you can explore.
+needs to handle signals you may need a more careful approach. Again, see
+``zc.async.subscribers`` for some options you can explore.
>>> import twisted.internet.selectreactor
>>> reactor = twisted.internet.selectreactor.SelectReactor()
@@ -362,42 +492,99 @@
>>> agent2.size
3
-We'll manipulate that a little later.
+We can change that at creation or later.
Finally, it's worth noting that agents contain the jobs that are currently
-be worked on by the dispatcher, on their behalf; and have a ``completed``
-collection of the more recent completed jobs, beginnin with the most recently
+worked on by the dispatcher, on their behalf; and have a ``completed``
+collection of the more recent completed jobs, beginning with the most recently
completed job.
----------------
-Multiple Queues
----------------
+----------------------
+Logging and Monitoring
+----------------------
-Since we put our queues in a mapping of them, we can also create multiple
-queues. This can make some scenarios more convenient and simpler to reason
-about. For instance, while you might have agents filtering jobs as we
-describe above, it might be simpler to say that you have a queue for one kind
-of job--say, processing a video file or an audio file--and a queue for other
-kinds of jobs. Then it is easy and obvious to set up simple FIFO agents
-as desired for different dispatchers. The same kind of logic could be
-accomplished with agents, but it is easier to picture the multiple queues.
+Logs are sent to the ``zc.async.events`` log for big events, like startup and
+shutdown, and errors. Poll and job logs are sent to ``zc.async.trace``.
+Confugure the standard Python logging module as usual to send these logs where
+you need. Be sure to auto-rotate the trace logs.
-------
-Quotas
-------
+The package supports monitoring using zc.z3monitor, but using this package
+includes more Zope 3 dependencies, so it is not included here. If you would
+like to use it, see monitor.txt and README_3.
-We touched on quotas in the usage section. Some jobs will need to
-access resoources that are shared across processes. A central data
-structure such as an index in the ZODB is a prime example, but other
-examples might include a network service that only allows a certain
-number of concurrent connections. These scenarios can be helped by
-quotas.
+ >>> reactor.stop()
-Quotas are demonstrated in the usage section. For configuration, you
-should know these characteristics:
+.. ......... ..
+.. Footnotes ..
+.. ......... ..
-- you cannot add a job with a quota name that is not defined in the queue;
+.. [#specific_dependencies] More specifically, as of this writing,
+ these are the minimal egg dependencies (including indirect
+ dependencies):
+ - pytz
+ A Python time zone library
+
+ - rwproperty
+ A small package of descriptor conveniences
+
+ - uuid
+ The uuid module included in Python 2.5
+
+ - zc.dict
+ A ZODB-aware dict implementation based on BTrees.
+
+ - zc.queue
+ A ZODB-aware queue
+
+ - zc.twist
+ Conveniences for working with Twisted and the ZODB
+
+ - twisted
+ The Twisted internet library.
+
+ - ZConfig
+ A general configuration package coming from the Zope project with which
+ the ZODB tests.
+
+ - zdaemon
+ A general daemon tool coming from the Zope project.
+
+ - ZODB3
+ The Zope Object Database.
+
+ - zope.bforest
+ Aggregations of multiple BTrees into a single dict-like structure,
+ reasonable for rotating data structures, among other purposes.
+
+ - zope.component
+ A way to hook together code by contract.
+
+ - zope.deferredimport
+ A way to defer imports in Python packages, often to prevent circular
+ import problems.
+
+ - zope.deprecation
+ A small framework for deprecating features.
+
+ - zope.event
+ An exceedingly small event framework that derives its power from
+ zope.component.
+
+ - zope.i18nmessageid
+ A way to specify strings to be translated.
+
+ - zope.interface
+ A way to specify code contracts and other data structures.
+
+ - zope.proxy
+ A way to proxy other Python objects.
+
+ - zope.testing
+ Testing extensions and helpers.
+
+.. [#undefined_quota_name]
+
>>> import operator
>>> import zc.async.job
>>> job = zc.async.job.Job(operator.mul, 5, 2)
@@ -411,8 +598,7 @@
>>> len(queue)
0
-- you cannot add a quota name to a job in a queue if the quota name is not
- defined in the queue;
+.. [#no_mutation_to_undefined]
>>> job.quota_names = ()
>>> job is queue.put(job)
@@ -424,7 +610,7 @@
>>> job.quota_names
()
-- you can create and remove quotas on the queue;
+.. [#create_remove_quotas]
>>> list(queue.quotas)
[]
@@ -435,8 +621,7 @@
>>> list(queue.quotas)
[]
-- you can remove quotas if pending jobs have their quota names--the quota name
- is then ignored;
+.. [#remove_quotas]
>>> queue.quotas.create('content catalog')
>>> job.quota_names = ('content catalog',)
@@ -448,13 +633,13 @@
>>> len(queue)
0
-- quotas default to a size of 1;
+.. [#default_size]
>>> queue.quotas.create('content catalog')
>>> queue.quotas['content catalog'].size
1
-- this can be changed at creation or later; and
+.. [#change_size]
>>> queue.quotas['content catalog'].size = 2
>>> queue.quotas['content catalog'].size
@@ -463,8 +648,7 @@
>>> queue.quotas['frobnitz account'].size
3
-- decreasing the size of a quota while the old quota size is filled will
- not affect the currently running jobs.
+.. [#decreasing_affects_future]
>>> job1 = zc.async.job.Job(operator.mul, 5, 2)
>>> job2 = zc.async.job.Job(operator.mul, 5, 2)
@@ -523,82 +707,6 @@
>>> quota.filled
False
-Additional Topics: Logging and Monitoring
-=========================================
-
-XXX see monitor.txt for sketch of zc.z3monitor monitoring.
-
- >>> reactor.stop()
-
-.. ......... ..
-.. Footnotes ..
-.. ......... ..
-
-.. [#specific_dependencies] More specifically, as of this writing,
- these are the minimal egg dependencies (including indirect
- dependencies):
-
- - pytz
- A Python time zone library
-
- - rwproperty
- A small package of descriptor conveniences
-
- - uuid
- The uuid module included in Python 2.5
-
- - zc.dict
- A ZODB-aware dict implementation based on BTrees.
-
- - zc.queue
- A ZODB-aware queue
-
- - zc.twist
- Conveniences for working with Twisted and the ZODB
-
- - twisted
- The Twisted internet library.
-
- - ZConfig
- A general configuration package coming from the Zope project with which
- the ZODB tests.
-
- - zdaemon
- A general daemon tool coming from the Zope project.
-
- - ZODB3
- The Zope Object Database.
-
- - zope.bforest
- Aggregations of multiple BTrees into a single dict-like structure,
- reasonable for rotating data structures, among other purposes.
-
- - zope.component
- A way to hook together code by contract.
-
- - zope.deferredimport
- A way to defer imports in Python packages, often to prevent circular
- import problems.
-
- - zope.deprecation
- A small framework for deprecating features.
-
- - zope.event
- An exceedingly small event framework that derives its power from
- zope.component.
-
- - zope.i18nmessageid
- A way to specify strings to be translated.
-
- - zope.interface
- A way to specify code contracts and other data structures.
-
- - zope.proxy
- A way to proxy other Python objects.
-
- - zope.testing
- Testing extensions and helpers.
-
.. [#get_poll]
>>> import time
Modified: zc.async/branches/dev/src/zc/async/agent.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/agent.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/agent.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -97,44 +97,15 @@
This particular agent invites you to provide a function to choose jobs.
The default one simply chooses the first available job in the queue.
-.. [#setUp]
+.. [#setUp] First we'll get a database and the necessary registrations.
>>> from ZODB.tests.util import DB
>>> db = DB()
>>> conn = db.open()
>>> root = conn.root()
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
- You must have two adapter registrations: IConnection to
- ITransactionManager, and IPersistent to IConnection. We will also
- register IPersistent to ITransactionManager because the adapter is
- designed for it.
-
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
-
- We need to be able to get data manager partials for functions and methods;
- normal partials for functions and methods; and a data manager for a partial.
- Here are the necessary registrations.
-
- >>> import zope.component
- >>> import types
- >>> import zc.async.interfaces
- >>> import zc.async.job
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
- ...
-
Now we need a queue.
>>> import zc.async.queue
@@ -146,10 +117,9 @@
Now we need an activated dispatcher agents collection.
- >>> import uuid
- >>> UUID = uuid.uuid1()
- >>> queue.dispatchers.register(UUID)
- >>> da = queue.dispatchers[UUID]
+ >>> import zc.async.instanceuuid
+ >>> queue.dispatchers.register(zc.async.instanceuuid.UUID)
+ >>> da = queue.dispatchers[zc.async.instanceuuid.UUID]
>>> da.activate()
And now we need an agent.
@@ -161,13 +131,6 @@
>>> agent.parent is da
True
- We need a UUID utility.
-
- >>> import zope.interface
- >>> zope.interface.classImplements(uuid.UUID, zc.async.interfaces.IUUID)
- >>> zope.component.provideUtility(
- ... UUID, zc.async.interfaces.IUUID, '')
-
.. [#test_completed]
>>> import zope.interface.verify
Added: zc.async/branches/dev/src/zc/async/configure.py
===================================================================
--- zc.async/branches/dev/src/zc/async/configure.py (rev 0)
+++ zc.async/branches/dev/src/zc/async/configure.py 2008-04-07 02:16:29 UTC (rev 85130)
@@ -0,0 +1,46 @@
+import types
+
+import zc.twist
+import zope.component
+import ZODB.interfaces
+
+import zc.async.interfaces
+import zc.async.job
+import zc.async.instanceuuid
+
+# These functions accomplish what configure.zcml does; you don't want both
+# to be in play (the component registry will complain).
+
+def minimal():
+ # use this ``minimal`` function if you have the
+ # zope.app.keyreference.persistent.connectionOfPersistent adapter
+ # installed in your zope.component registry. Otherwise use ``base``
+ # below.
+
+ # persistent object and connection -> transaction manager
+ zope.component.provideAdapter(zc.twist.transactionManager)
+ zope.component.provideAdapter(zc.twist.transactionManager,
+ adapts=(ZODB.interfaces.IConnection,))
+
+ # function and method -> job
+ zope.component.provideAdapter(
+ zc.async.job.Job,
+ adapts=(types.FunctionType,),
+ provides=zc.async.interfaces.IJob)
+ zope.component.provideAdapter(
+ zc.async.job.Job,
+ adapts=(types.MethodType,),
+ provides=zc.async.interfaces.IJob)
+ zope.component.provideAdapter( # optional, rarely used
+ zc.async.job.Job,
+ adapts=(zc.twist.METHOD_WRAPPER_TYPE,),
+ provides=zc.async.interfaces.IJob)
+
+ # UUID for this instance
+ zope.component.provideUtility(
+ zc.async.instanceuuid.UUID, zc.async.interfaces.IUUID)
+
+def base():
+ # see comment in ``minimal``, above
+ minimal()
+ zope.component.provideAdapter(zc.twist.connection)
\ No newline at end of file
Modified: zc.async/branches/dev/src/zc/async/dispatcher.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/dispatcher.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/dispatcher.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -50,7 +50,9 @@
Here's the reactor. The _handleSignals just lets the reactor handle signals.
In most real world usage you'll need to be more careful, hooking into
-the signal handling of your larger app.
+the signal handling of your larger app. Look at the code in
+``zc.async.subscribers.ThreadedDispatcherInstaller`` for an example that should
+be sufficient for Zope.
>>> import twisted.internet.selectreactor
>>> reactor = twisted.internet.selectreactor.SelectReactor()
@@ -63,7 +65,8 @@
>>> import zc.async.dispatcher
>>> dispatcher = zc.async.dispatcher.Dispatcher(
... db, reactor, poll_interval=0.5)
- >>> dispatcher.UUID is UUID
+ >>> import zc.async.instanceuuid
+ >>> dispatcher.UUID is zc.async.instanceuuid.UUID
True
>>> dispatcher.reactor is reactor
True
@@ -336,13 +339,7 @@
When we stop the reactor, the dispatcher also deactivates.
>>> reactor.callFromThread(reactor.stop)
- >>> for i in range(30):
- ... if not dispatcher.activated:
- ... break
- ... time.sleep(0.1)
- ... else:
- ... assert False, 'dispatcher did not deactivate'
- ...
+ >>> thread.join(3)
>>> pprint.pprint(dispatcher.getStatusInfo()) # doctest: +ELLIPSIS
{'poll interval': datetime.timedelta(0, 0, 500000),
@@ -369,44 +366,9 @@
>>> db = DB(storage)
>>> conn = db.open()
>>> root = conn.root()
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
- You must have two adapter registrations: IConnection to
- ITransactionManager, and IPersistent to IConnection. We will also
- register IPersistent to ITransactionManager because the adapter is
- designed for it.
-
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
-
- We need to be able to get data manager partials for functions and methods;
- normal partials for functions and methods; and a data manager for a partial.
- Here are the necessary registrations.
-
- >>> import zope.component
- >>> import types
- >>> import zc.async.interfaces
- >>> import zc.async.job
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
- ...
-
- Now we need a UUID. We'll use the instanceuuid.
-
- >>> from zc.async.instanceuuid import UUID
- >>> zope.component.provideUtility(
- ... UUID, zc.async.interfaces.IUUID, '')
-
.. [#get_poll]
>>> import time
Modified: zc.async/branches/dev/src/zc/async/job.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/job.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/job.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -772,42 +772,16 @@
=========
.. [#set_up] We'll actually create the state that the text needs here.
+ One thing to notice is that the ``zc.async.configure.base`` registers
+ the Job class as an adapter from functions and methods.
>>> from ZODB.tests.util import DB
>>> db = DB()
>>> conn = db.open()
>>> root = conn.root()
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
- You must have two adapter registrations: IConnection to
- ITransactionManager, and IPersistent to IConnection. We will also
- register IPersistent to ITransactionManager because the adapter is
- designed for it.
-
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
-
- The job class can be registered as an adapter for
- functions and methods. It needs to be for expected simple usage of
- addCallbacks.
-
- >>> import zope.component
- >>> import types
- >>> import zc.async.interfaces
- >>> import zc.async.job
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
-
.. [#verify] Verify interface
>>> from zope.interface.verify import verifyObject
Modified: zc.async/branches/dev/src/zc/async/jobs_and_transactions.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/jobs_and_transactions.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/jobs_and_transactions.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -264,31 +264,5 @@
>>> db = DB()
>>> conn = db.open()
>>> root = conn.root()
-
- You must have two adapter registrations: IConnection to
- ITransactionManager, and IPersistent to IConnection. We will also
- register IPersistent to ITransactionManager because the adapter is
- designed for it.
-
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
-
- We also need to adapt Function and Method to IPartial.
-
- >>> import zc.async.job
- >>> import zc.async.interfaces
- >>> import zope.component
- >>> import types
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
\ No newline at end of file
Modified: zc.async/branches/dev/src/zc/async/monitor.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/monitor.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/monitor.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -283,32 +283,6 @@
.. [#setUp] See the discussion in other documentation to explain this code.
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
-
- >>> import zope.component
- >>> import types
- >>> import zc.async.interfaces
- >>> import zc.async.job
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
- ...
-
- >>> import zc.async.testing
- >>> reactor = zc.async.testing.Reactor()
- >>> reactor.start() # this mokeypatches datetime.datetime.now
-
>>> import ZODB.FileStorage
>>> storage = ZODB.FileStorage.FileStorage(
... 'zc_async.fs', create=True)
@@ -317,6 +291,13 @@
>>> conn = db.open()
>>> root = conn.root()
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
+
+ >>> import zc.async.testing
+ >>> reactor = zc.async.testing.Reactor()
+ >>> reactor.start() # this mokeypatches datetime.datetime.now
+
>>> import zc.async.queue
>>> import zc.async.interfaces
>>> mapping = root[zc.async.interfaces.KEY] = zc.async.queue.Queues()
@@ -324,11 +305,6 @@
>>> import transaction
>>> transaction.commit()
- >>> from zc.async.instanceuuid import UUID
- >>> import zope.component
- >>> zope.component.provideUtility(
- ... UUID, zc.async.interfaces.IUUID, '')
-
>>> import zc.async.dispatcher
>>> dispatcher = zc.async.dispatcher.Dispatcher(db, reactor)
>>> dispatcher.activate()
Modified: zc.async/branches/dev/src/zc/async/queue.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/queue.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/queue.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -62,11 +62,8 @@
callable. This code also expects that a UUID will be registered as a
zc.async.interfaces.IUUID utility with the empty name ('').
- >>> import uuid
- >>> UUID = uuid.uuid1()
+ >>> from zc.async.instanceuuid import UUID
-[#setUp_UUID_utility]_
-
>>> import zc.async.testing
>>> zc.async.testing.setUpDatetime()
@@ -366,6 +363,7 @@
Now I can claim job4 and put in a (stub) agent. Until job4 has moved to the
CALLBACKS or COMPLETED status, I will be unable to claim job4.
+ >>> import zope.interface
>>> import persistent.list
>>> import BTrees
>>> class Completed(persistent.Persistent):
@@ -453,8 +451,8 @@
Dispatchers typically get their UUID from the instanceuuid module in
this package, but we will generate our own here.
-First we'll register dispatcher using the UUID we created near the beginning
-of this document.
+First we'll register dispatcher using the instance UUID we introduced near the
+beginning of this document.
>>> UUID in queue.dispatchers
False
@@ -693,6 +691,7 @@
We'll introduce another virtual dispatcher to show this behavior.
+ >>> import uuid
>>> alt_UUID = uuid.uuid1()
>>> queue.dispatchers.register(alt_UUID)
>>> alt_da = queue.dispatchers[alt_UUID]
@@ -776,38 +775,9 @@
>>> db = DB()
>>> conn = db.open()
>>> root = conn.root()
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
- You must have two adapter registrations: IConnection to
- ITransactionManager, and IPersistent to IConnection. We will also
- register IPersistent to ITransactionManager because the adapter is
- designed for it.
-
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
-
- We need to be able to get data manager partials for functions and methods;
- normal partials for functions and methods; and a data manager for a partial.
- Here are the necessary registrations.
-
- >>> import zope.component
- >>> import types
- >>> import zc.async.interfaces
- >>> import zc.async.job
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
- ...
-
.. [#queues_collection] The queues collection is a simple mapping that only
allows queues to be inserted.
@@ -863,16 +833,6 @@
>>> verifyObject(zc.async.interfaces.IQueue, queue)
True
-.. [#setUp_UUID_utility] We need to provide an IUUID utility that
- identifies the current instance.
-
- >>> import uuid
- >>> zope.interface.classImplements(uuid.UUID, zc.async.interfaces.IUUID)
- >>> zope.component.provideUtility(
- ... UUID, zc.async.interfaces.IUUID, '')
-
- Normally this would be the UUID instance in zc.async.instanceuuid.
-
.. [#check_dispatchers_mapping]
>>> len(queue.dispatchers)
Modified: zc.async/branches/dev/src/zc/async/subscribers.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/subscribers.txt 2008-04-06 23:41:18 UTC (rev 85129)
+++ zc.async/branches/dev/src/zc/async/subscribers.txt 2008-04-07 02:16:29 UTC (rev 85130)
@@ -190,32 +190,9 @@
>>> db.database_name = ''
>>> async_db.database_name = 'async'
- >>> from zc.twist import transactionManager, connection
- >>> import zope.component
- >>> zope.component.provideAdapter(transactionManager)
- >>> zope.component.provideAdapter(connection)
- >>> import ZODB.interfaces
- >>> zope.component.provideAdapter(
- ... transactionManager, adapts=(ZODB.interfaces.IConnection,))
+ >>> import zc.async.configure
+ >>> zc.async.configure.base()
- >>> import zope.component
- >>> import types
- >>> import zc.async.interfaces
- >>> import zc.async.job
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.FunctionType,),
- ... provides=zc.async.interfaces.IJob)
- >>> zope.component.provideAdapter(
- ... zc.async.job.Job,
- ... adapts=(types.MethodType,),
- ... provides=zc.async.interfaces.IJob)
- ...
-
- >>> from zc.async.instanceuuid import UUID
- >>> zope.component.provideUtility(
- ... UUID, zc.async.interfaces.IUUID, '')
-
>>> import time
>>> def get_poll(count = None):
... if count is None:
@@ -226,15 +203,4 @@
... time.sleep(0.1)
... else:
... assert False, 'no poll!'
- ...
-
- >>> import zc.async.interfaces
- >>> def wait_for_result(job):
- ... for i in range(30):
- ... t = transaction.begin()
- ... if job.status == zc.async.interfaces.COMPLETED:
- ... return job.result
- ... time.sleep(0.1)
- ... else:
- ... assert False, 'job never completed'
- ...
\ No newline at end of file
+ ...
More information about the Checkins
mailing list