[Checkins] SVN: zc.async/trunk/src/zc/async/ hook up S5 quickstart
as test of configure.
Gary Poster
gary at zope.com
Wed Jul 30 16:08:45 EDT 2008
Log message for revision 89065:
hook up S5 quickstart as test of configure.
Changed:
U zc.async/trunk/src/zc/async/QUICKSTART_1_VIRTUALENV.txt
U zc.async/trunk/src/zc/async/configure.py
U zc.async/trunk/src/zc/async/tests.py
-=-
Modified: zc.async/trunk/src/zc/async/QUICKSTART_1_VIRTUALENV.txt
===================================================================
--- zc.async/trunk/src/zc/async/QUICKSTART_1_VIRTUALENV.txt 2008-07-30 19:27:25 UTC (rev 89064)
+++ zc.async/trunk/src/zc/async/QUICKSTART_1_VIRTUALENV.txt 2008-07-30 20:08:44 UTC (rev 89065)
@@ -62,12 +62,14 @@
``zc.async.job.parallel``. Given three decomposed tasks, ``job_A``,
``job_B``, and ``job_C``; a postprocess task named ``postprocess``; and an
instance of a zc.async queue, this line would schedule a composite parallel
- job:
+ job::
- >>> queue.put(
+ >> queue.put(
... zc.async.job.parallel(
... job_A, job_B, job_C, postprocess=postprocess))
+.. We use ">>" intentionally when we don't want to run these lines as a test.
+
...for serial processing
========================
@@ -78,9 +80,9 @@
As shown later, the easiest way to accomplish this is to use
``zc.async.job.serial``. Given three decomposed tasks, ``job_A``, ``job_B``,
and ``job_C``; a postprocess task named ``postprocess``; and an instance
- of a zc.async queue, this line would schedule a composite serial job:
+ of a zc.async queue, this line would schedule a composite serial job::
- >>> queue.put(
+ >> queue.put(
... zc.async.job.serial(
... job_A, job_B, job_C, postprocess=postprocess))
@@ -113,9 +115,14 @@
An Experiment
=============
-To start, install ``virtualenv``_ and create a virtual environment for our
+To start, install |virtualenv|_ and create a virtual environment for our
experiments.
+.. class:: handout
+
+ I prefer zc.buildout for production deployments, but virtualenv is very
+ nice for quick experimentation.
+
Install zc.async in the virtual environment.
::
@@ -125,6 +132,8 @@
$ cd quickstart/
$ ./bin/easy_install zc.async
+.. |virtualenv| replace:: ``virtualenv``
+
.. _virtualenv: http://pypi.python.org/pypi/virtualenv
Dependencies
@@ -167,10 +176,10 @@
ZEO Server
==========
-zc.async relies on ZEO ("Zope Enterprise Objects") to distribute work.
+zc.async relies on a distributed ZODB technology called ZEO ("Zope Enterprise
+Objects") to distribute work. ZEO has a central database server to which client
+processes connect.
-ZEO has a central database server to which client processes connect.
-
Let's start the ZEO Server::
./bin/runzeo -a 9999 -f test.fs &
@@ -188,33 +197,51 @@
$ ./bin/python
-Make your database connection.
+This will be our single client process. You might have many.
- >>> import ZEO.ClientStorage
- >>> import ZODB
- >>> storage = ZEO.ClientStorage.ClientStorage(('127.0.0.1', 9999))
- >>> db = ZODB.DB(storage)
+Database Connection
+===================
-Start zc.async
-==============
+Connect to the database.
+::
+
+ >> import ZEO.ClientStorage
+ >> import ZODB
+ >> storage = ZEO.ClientStorage.ClientStorage(
+ ... ('127.0.0.1', 9999))
+ >> db = ZODB.DB(storage)
+
+.. When run as a doctest, this uses a simple FileStorage, rather than a
+ ClientStorage.
+
+ >>> import ZODB.FileStorage
+ >>> storage = ZODB.FileStorage.FileStorage(
+ ... 'zc_async.fs', create=True)
+ >>> from ZODB.DB import DB
+ >>> db = DB(storage)
+
+Start zc.async: Basics
+======================
+
Now we do some basic configuration. This first bit installs some default
adapters. You might not ever have to worry too much about them.
>>> import zc.async.configure
>>> zc.async.configure.base()
+Start zc.async: Policy
+======================
+
This second part is policy, and if you ever put zc.async in production, you'll
want to understand what's going on here. We'll talk about what's going on here
a little later.
- >>> zc.async.configure.start(db, poll_interval=1)
+ >>> zc.async.configure.start(
+ ... db, poll_interval=1)
-The ``poll_interval`` argument means that this instance will poll for new jobs
-every one second.
+Now the system is polling for jobs every second.
-Now the system is polling for jobs.
-
The Queue
=========
@@ -231,15 +258,20 @@
A Job
=====
-Let's put a job in our queue. This is a silly example. Imagine instead that
-this was some really long-running job. Maybe you have lots of these jobs
-coming in, and you need to have many machines to claim jobs and perform them,
-so that you can scale. Maybe this job divides itself up into parallel or
-serial jobs, and this parent job isn't done until all the children jobs run to
-completion.
+Let's put a job in our queue.
-Or maybe this is a silly example.
+.. class:: handout
+ This is a silly example. Imagine instead that this was some really
+ long-running job. Maybe you have lots of these jobs coming in, and you need
+ to have many machines to claim jobs and perform them, so that you can
+ scale. Maybe this job divides itself up into parallel or serial jobs, and
+ this parent job isn't done until all the children jobs run to completion.
+
+ Or maybe this is a silly example.
+
+..
+
>>> import time
>>> j = q.put(time.time)
>>> j.result
@@ -257,8 +289,14 @@
Now wait a second and then try this. "transaction.begin" will sync up our
database with database changes made elsewhere.
- >>> transaction.begin()
- <transaction._transaction.Transaction object at 0x163ad30>
+.. This lets us "wait a second".
+
+ >>> import zc.async.testing
+ >>> res = zc.async.testing.wait_for_result(j)
+
+..
+
+ >>> _ = transaction.begin()
>>> j.result
1216179006.856108
>>> j.status
@@ -270,36 +308,26 @@
You can also make closures by passing in the job class explicitly. Generating
RSA keys is actually a reasonable real-world use case for something like this.
- >>> import subprocess
- >>> j = q.put(zc.async.job.Job(
- ... subprocess.call, ['openssl', 'genrsa', '-out', 'key.pem', '1024']))
- >>> transaction.commit()
+::
+ >> import subprocess
+ >> j = q.put(zc.async.job.Job(
+ ... subprocess.call,
+ ... ['openssl', 'genrsa', '-out',
+ ... 'key.pem', '1024']))
+ >> transaction.commit()
+
Another Result
==============
- >>> j.result
- >>> transaction.begin()
- <transaction._transaction.Transaction object at 0x15b86b0>
- >>> j.result
+ >> j.result
+ >> _ = transaction.begin()
+ >> j.result
0
- >>> subprocess.call(['ls'])
- bin key.pem test.fs test.fs.lock uuid.txt
- include lib test.fs.index test.fs.tmp
- >>> subprocess.call(['cat', 'key.pem'])
+ >> subprocess.call(['cat', 'key.pem'])
-----BEGIN RSA PRIVATE KEY-----
MIICXgIBAAKBgQCYAZW+HjDGJhRHnUlZZWqhrGOxU2K/RhssmcMs0JLnWI2cWmZ+
- 3S1LxlUqTiPcbzqkgyls13+L7jCYkMNVeGwblL/t1qSRBeOEVlOXzB2UQEQ8dDXm
- FszLaFJ+nuFBYjCTl/sPkclrzk1AQX6vEji82kJG2GUPOyj3owoMN0AF7wIDAQAB
- AoGAPVfmZmMoq86SQJRpFXqoHbbERLuyDh7suIMVEPDbWCGUhJS26Fu5++p+VCRU
- tJDuyZVlElelUYM+eVNygPuoJJoaHHl34CS1/Qu5YC2XgNTt5dAUxShU6+M9puGA
- Kc2x6QwwBCHlCe753MK+VV5RGXgNC/7xNHV0IK3/azD+ZQECQQDGsOv13ux7u4aQ
- jpNYGgF+iQJyEj67uFlDI395yCw+HSuXXOat/bJLOgSJfOnsHnmsY1BdEu44Q4ON
- u/ShHlkHAkEAw9mDeDFW0/hGPC7VTia+dvEEcWNo0P5CNiB+67pY/qOGYBW3WgcU
- iWNXoNLrvm6XVwDXFPqYMlu292oI3qw52QJBAJ6FQwy8GZKyT67/gYD15qFMsF3Q
- PqrIbrcJGEhSMzIvVbsCjKzeTqSEGmCS/5K50bt+1PwdAWB0RP4MqiTtsHsCQQCr
- aF2F/jiuACcIWTzaz1H3K23mB0kfUMiGMt6iVU+6XUgoJBl6s6OnsshouvEUlBnk
- TZnwhHpb6KUz2Ru2NynRAkEAtjnNxNcmFDSOTyYan8PIZ1x0trDWy5hYnC9YD0Ts
+ ...
CEcz6ZbO8zm4AEGI/dqLicZh3bhunhflAovW6WxbNKLENQ==
-----END RSA PRIVATE KEY-----
0
@@ -320,3 +348,11 @@
Close by referring to production instances needing something like zdaemon
or supervisor; and to preferring the more declarative zc.buildout style for
production...which we'll show in our next quickstart! ;-)
+
+.. Now we are going to stop the reactor.
+
+ >>> import zc.async.dispatcher
+ >>> dispatcher = zc.async.dispatcher.get()
+ >>> reactor = dispatcher.reactor
+ >>> reactor.callFromThread(reactor.stop)
+ >>> dispatcher.thread.join(3)
Modified: zc.async/trunk/src/zc/async/configure.py
===================================================================
--- zc.async/trunk/src/zc/async/configure.py 2008-07-30 19:27:25 UTC (rev 89064)
+++ zc.async/trunk/src/zc/async/configure.py 2008-07-30 20:08:44 UTC (rev 89065)
@@ -71,8 +71,10 @@
# this function installs a queue named '' (empty string), starts the
# dispatcher, and installs an agent named 'main', with default values.
# It is a convenience for quick starts.
-def start(db, poll_interval=5, db_name=None, chooser=None, size=3):
+def start(db, poll_interval=5, db_name=None, agent_chooser=None, agent_size=3):
zope.component.provideAdapter(zc.async.queue.getDefaultQueue)
+ zope.component.provideAdapter(zc.async.queue.getDefaultQueue,
+ adapts=(ZODB.interfaces.IConnection,))
zope.component.provideHandler(
zc.async.subscribers.QueueInstaller(db_name=db_name))
zope.component.provideHandler(
Modified: zc.async/trunk/src/zc/async/tests.py
===================================================================
--- zc.async/trunk/src/zc/async/tests.py 2008-07-30 19:27:25 UTC (rev 89064)
+++ zc.async/trunk/src/zc/async/tests.py 2008-07-30 20:08:44 UTC (rev 89065)
@@ -13,8 +13,9 @@
##############################################################################
import os
import unittest
+import re
-from zope.testing import doctest, module, loggingsupport
+from zope.testing import doctest, module, loggingsupport, renormalizing
import zope.component
import zope.component.testing
import zope.component.eventtesting
@@ -70,12 +71,12 @@
"""This module provides access to a UUID that is intended to uniquely
identify this software instance. Read the `msg` value below for more
information.
-
+
The uuid is generated and then stashed in a file. It only works if
the INSTANCE_HOME environment variable is set to a folder that has an
`etc` folder in it--a standard Zope set up. For this test, we mock it
up in uuidSetUp and uuidTearDown below.
-
+
>>> import zc.async.instanceuuid
>>> import uuid
>>> isinstance(zc.async.instanceuuid.UUID, uuid.UUID)
@@ -86,7 +87,7 @@
True
uuid.UUIDs now provide zc.async.interfaces.IUUID
-
+
>>> import zc.async.interfaces
>>> zc.async.interfaces.IUUID.implementedBy(uuid.UUID)
True
@@ -96,7 +97,7 @@
That's a bit invasive, but now you can register the instance UUID as
a utility and get it back out as something that provides
zc.async.interfaces.IUUID.
-
+
>>> import zope.component
>>> zope.component.provideUtility(
... zc.async.instanceuuid.UUID, name='instance')
@@ -114,7 +115,7 @@
and back again. Dates in the future get smaller and smaller, so
dates are arranged from newest to oldest in a BTree. It leaves an
extra 4 bits at the bottom. It can convert all possible datetimes.
-
+
>>> from zc.async.utils import long_to_dt, dt_to_long
>>> import datetime
>>> now = datetime.datetime.now()
@@ -131,6 +132,7 @@
True
"""
+
def test_suite():
return unittest.TestSuite((
doctest.DocTestSuite(setUp=uuidSetUp, tearDown=uuidTearDown),
@@ -145,8 +147,11 @@
'README_2.txt',
'catastrophes.txt',
'ftesting.txt',
+ 'QUICKSTART_1_VIRTUALENV.txt',
setUp=modSetUp, tearDown=modTearDown,
- optionflags=doctest.INTERPRET_FOOTNOTES),
+ optionflags=doctest.INTERPRET_FOOTNOTES,
+ checker = renormalizing.RENormalizing([ # used by QUICKSTART only
+ (re.compile('\d+\.\d+'), '1216179006.856108')])),
))
More information about the Checkins
mailing list