[Checkins] SVN: zc.async/branches/dev/src/zc/async/ big effort is
README_3, configuration in Zope 3. Also set up for log tests;
TODO notes for log optimizations.
Gary Poster
gary at zope.com
Mon Apr 7 23:25:12 EDT 2008
Log message for revision 85155:
big effort is README_3, configuration in Zope 3. Also set up for log tests; TODO notes for log optimizations.
Changed:
A zc.async/branches/dev/src/zc/async/README_3.txt
U zc.async/branches/dev/src/zc/async/TODO.txt
U zc.async/branches/dev/src/zc/async/dispatcher.py
A zc.async/branches/dev/src/zc/async/multidb_dispatcher_policy.zcml
U zc.async/branches/dev/src/zc/async/queue.py
U zc.async/branches/dev/src/zc/async/tests.py
U zc.async/branches/dev/src/zc/async/z3tests.py
-=-
Added: zc.async/branches/dev/src/zc/async/README_3.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/README_3.txt (rev 0)
+++ zc.async/branches/dev/src/zc/async/README_3.txt 2008-04-08 03:25:09 UTC (rev 85155)
@@ -0,0 +1,282 @@
+=========================
+Configuration With Zope 3
+=========================
+
+Our last main section can be the shortest yet, both because we've already
+introduced all of the main concepts, and because we will be leveraging
+conveniences to automate much of the configuration shown in the section
+discussing configuration without Zope 3.
+
+If you want to set up a client alone, without a dispatcher, include
+configure.zcml, make sure you share the database in which the queues will be
+held, and make sure that either the
+zope.app.keyreference.persistent.connectionOfPersistent adapter is registered,
+or zc.twist.connection.
+
+For a client/server combination, use zcml that is something like the
+basic_dispatcher_policy.zcml, make sure you have access to the database with
+the queues, configure logging and monitoring as desired, configure the
+ZC_ASYNC_UUID environmental variable in zdaemon.conf if you are in production,
+and start up! Getting started is really pretty easy. You can even start a
+dispatcher-only version by not starting any servers in zcml.
+
+We'll look at this by making a zope.conf-alike and a site.zcml-alike. We'll
+need a place to put some files, so we'll use a temporary directory. This, and
+the comments in the files that we set up. are the primary differences between
+our examples and a real set up.
+
+So, without further ado, here is the text of our zope.conf-alike, and of our
+site.zcml-alike [#get_vals]_. We'll be using two databases for this example,
+as you might want for a site with a fair amount of zc.async usage.
+
+ >>> zope_conf = """
+ ... site-definition %(site_zcml_file)s
+ ...
+ ... <zodb main>
+ ... <filestorage>
+ ... create true
+ ... path %(main_storage_path)s
+ ... </filestorage>
+ ... </zodb>
+ ...
+ ... <zodb async>
+ ... <filestorage>
+ ... create true
+ ... path %(async_storage_path)s
+ ... </filestorage>
+ ... </zodb>
+ ...
+ ... <product-config zc.z3monitor>
+ ... port %(monitor_port)s
+ ... </product-config>
+ ...
+ ... <logger>
+ ... level debug
+ ... name zc.async
+ ... propagate no
+ ...
+ ... <logfile>
+ ... path %(async_event_log)s
+ ... </logfile>
+ ... </logger>
+ ...
+ ... <logger>
+ ... level debug
+ ... name zc.async.trace
+ ... propagate no
+ ...
+ ... <logfile>
+ ... path %(async_trace_log)s
+ ... </logfile>
+ ... </logger>
+ ...
+ ... <eventlog>
+ ... <logfile>
+ ... formatter zope.exceptions.log.Formatter
+ ... path STDOUT
+ ... </logfile>
+ ... <logfile>
+ ... formatter zope.exceptions.log.Formatter
+ ... path %(event_log)s
+ ... </logfile>
+ ... </eventlog>
+ ... """ % {'site_zcml_file': site_zcml_file,
+ ... 'main_storage_path': os.path.join(dir, 'main.fs'),
+ ... 'async_storage_path': os.path.join(dir, 'async.fs'),
+ ... 'monitor_port': monitor_port,
+ ... 'event_log': os.path.join(dir, 'z3.log'),
+ ... 'async_event_log': os.path.join(dir, 'async.log'),
+ ... 'async_trace_log': os.path.join(dir, 'async_trace.log'),}
+ ...
+
+In a non-trivial production system of you will also probably want to replace
+the two file storages with two <zeoclient> stanzas.
+
+Also note that an open monitor port should be behind a firewall, of course.
+
+We'll assume that zdaemon.conf has been set up to put ZC_ASYNC_UUID in the
+proper place too. It would have looked something like this in the
+zdaemon.conf::
+
+ <environment>
+ ZC_ASYNC_UUID /Users/gary/opt/classifieds/parts/classifieds/uuid.txt
+ </environment>
+
+(Other tools, such as supervisor, also can work, of course; their spellings are
+different and are "left as an exercise to the reader" at the moment.)
+
+We'll do that "by hand":
+
+ >>> os.environ['ZC_ASYNC_UUID'] = os.path.join(dir, 'uuid.txt')
+
+Now let's define our site-zcml-alike.
+
+ >>> site_zcml = """
+ ... <configure xmlns='http://namespaces.zope.org/zope'
+ ... xmlns:meta="http://namespaces.zope.org/meta"
+ ... >
+ ... <include package="zope.component" file="meta.zcml" />
+ ... <include package="zope.component" />
+ ... <include package="zc.async" file="multidb_dispatcher_policy.zcml" />
+ ...
+ ... <!-- this is usually handled in Zope applications by the
+ ... zope.app.keyreference.persistent.connectionOfPersistent adapter -->
+ ... <adapter factory="zc.twist.connection" />
+ ... </configure>
+ ... """
+
+Now we're done.
+
+If you want to change policy, change "multidb_dispatcher_policy.zcml" to
+"dispatcher.zcml" in the example above and register your replacement bits for
+the policy in "multidb_dispatcher_policy.zcml". You'll see that most of that
+comes from code in subscribers.py, which can be adjusted easily.
+
+If we process these files, and wait for a poll, we've got a working
+set up[#process]_.
+
+ >>> import zc.async.dispatcher
+ >>> dispatcher = zc.async.dispatcher.get()
+ >>> import pprint
+ >>> pprint.pprint(get_poll(0))
+ {'': {'main': {'active jobs': [],
+ 'error': None,
+ 'len': 0,
+ 'new jobs': [],
+ 'size': 3}}}
+ >>> bool(dispatcher.activated)
+ True
+
+We can ask for a job to be performed, and get the result.
+
+ >>> conn = db.open()
+ >>> root = conn.root()
+ >>> import zc.async.interfaces
+ >>> queue = zc.async.interfaces.IQueue(root)
+ >>> import operator
+ >>> import zc.async.job
+ >>> job = queue.put(zc.async.job.Job(operator.mul, 21, 2))
+ >>> import transaction
+ >>> transaction.commit()
+ >>> wait_for_result(job)
+ 42
+
+TODO look at files, maybe even use telnetlib to show that you can get over to
+the monitor port, for amusement.
+
+Now we'll "shut down" with a CTRL-C, or SIGINT.
+
+ >>> import signal
+ >>> if getattr(os, 'getpid', None) is not None: # UNIXEN, not Windows
+ ... pid = os.getpid()
+ ... try:
+ ... os.kill(pid, signal.SIGINT)
+ ... except KeyboardInterrupt:
+ ... if dispatcher.activated:
+ ... assert False, 'dispatcher did not deactivate'
+ ... else:
+ ... print "failed to send SIGINT, or something"
+ ... else:
+ ... dispatcher.reactor.callFromThread(dispatcher.reactor.stop)
+ ... for i in range(30):
+ ... if not dispatcher.activated:
+ ... break
+ ... time.sleep(0.1)
+ ... else:
+ ... assert False, 'dispatcher did not deactivate'
+ ...
+ >>> import transaction
+ >>> t = transaction.begin() # sync
+ >>> import zope.component
+ >>> import zc.async.interfaces
+ >>> uuid = zope.component.getUtility(zc.async.interfaces.IUUID)
+ >>> da = queue.dispatchers[uuid]
+ >>> bool(da.activated)
+ False
+
+Now we'll clean up.
+
+ >>> db.close()
+ >>> db.databases['async'].close()
+ >>> import shutil
+ >>> shutil.rmtree(dir)
+
+.. ......... ..
+.. Footnotes ..
+.. ......... ..
+
+.. [#get_vals]
+
+ >>> import errno, os, random, socket, tempfile
+ >>> dir = tempfile.mkdtemp()
+ >>> site_zcml_file = os.path.join(dir, 'site.zcml')
+
+ >>> s = socket.socket()
+ >>> for i in range(20):
+ ... monitor_port = random.randint(20000, 49151)
+ ... try:
+ ... s.bind(('127.0.0.1', monitor_port))
+ ... except socket.error, e:
+ ... if e.args[0] == errno.EADDRINUSE:
+ ... pass
+ ... else:
+ ... raise
+ ... else:
+ ... s.close()
+ ... break
+ ... else:
+ ... assert False, 'could not find available port'
+ ... monitor_port = None
+ ...
+
+.. [#process]
+
+ >>> zope_conf_file = os.path.join(dir, 'zope.conf')
+ >>> f = open(zope_conf_file, 'w')
+ >>> f.write(zope_conf)
+ >>> f.close()
+ >>> f = open(site_zcml_file, 'w')
+ >>> f.write(site_zcml)
+ >>> f.close()
+
+ >>> import zdaemon.zdoptions
+ >>> import zope.app.appsetup
+ >>> options = zdaemon.zdoptions.ZDOptions()
+ >>> options.schemadir = os.path.join(
+ ... os.path.dirname(os.path.abspath(zope.app.appsetup.__file__)),
+ ... 'schema')
+ >>> options.realize(['-C', zope_conf_file])
+ >>> config = options.configroot
+
+ >>> import zope.app.appsetup.product
+ >>> zope.app.appsetup.product.setProductConfigurations(
+ ... config.product_config)
+ >>> ignore = zope.app.appsetup.config(config.site_definition)
+ >>> import zope.app.appsetup.appsetup
+ >>> db = zope.app.appsetup.appsetup.multi_database(config.databases)[0][0]
+
+ >>> import zope.event
+ >>> import zc.async.interfaces
+ >>> zope.event.notify(zc.async.interfaces.DatabaseOpened(db))
+
+ >>> import time
+ >>> def get_poll(count = None):
+ ... if count is None:
+ ... count = len(dispatcher.polls)
+ ... for i in range(30):
+ ... if len(dispatcher.polls) > count:
+ ... return dispatcher.polls.first()
+ ... time.sleep(0.1)
+ ... else:
+ ... assert False, 'no poll!'
+ ...
+
+ >>> def wait_for_result(job):
+ ... for i in range(30):
+ ... t = transaction.begin()
+ ... if job.status == zc.async.interfaces.COMPLETED:
+ ... return job.result
+ ... time.sleep(0.5)
+ ... else:
+ ... assert False, 'job never completed'
+ ...
\ No newline at end of file
Property changes on: zc.async/branches/dev/src/zc/async/README_3.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Modified: zc.async/branches/dev/src/zc/async/TODO.txt
===================================================================
--- zc.async/branches/dev/src/zc/async/TODO.txt 2008-04-07 16:01:47 UTC (rev 85154)
+++ zc.async/branches/dev/src/zc/async/TODO.txt 2008-04-08 03:25:09 UTC (rev 85155)
@@ -1,12 +1,10 @@
-
+- make trace log only write changes or occasionally
+- make data structure for old polls more efficient
+- make period for polls and jobs shorter
+- make failures reduce to small size (eliminate stack?)
+- make full tracebacks write to log, or at least an option
- Write the z3monitor tests.
- Write a stress test.
-- Finish the README_2 docs.
-- Write a way to use alone, with a separate zdaemon script.
-- Write a way to use with Zope 3 (in particular, something that waits for the
- database opened event that Zope 3 fires).
-- remove subscribers.py, or make it part of the Zope 3 bit above.
-- write basic zcml (necessary adapters) and full (more policy) zcml
For future versions:
Modified: zc.async/branches/dev/src/zc/async/dispatcher.py
===================================================================
--- zc.async/branches/dev/src/zc/async/dispatcher.py 2008-04-07 16:01:47 UTC (rev 85154)
+++ zc.async/branches/dev/src/zc/async/dispatcher.py 2008-04-08 03:25:09 UTC (rev 85155)
@@ -144,14 +144,21 @@
# should come before 'completed' for threading dance
if isinstance(job.result, twisted.python.failure.Failure):
info['failed'] = True
+ info['result'] = job.result.getTraceback(
+ elideFrameworkCode=True, detail='verbose')
+ else:
+ info['result'] = repr(job.result)
info['completed'] = datetime.datetime.utcnow()
- info['result'] = repr(job.result)
finally:
local.job = None
transaction.abort()
conn.close()
- zc.async.utils.tracelog.info(
- '%s %s in thread %d with result %s',
+ if info['failed']:
+ log = zc.async.utils.tracelog.error
+ else:
+ log = zc.async.utils.tracelog.info
+ log(
+ '%s %s in thread %d with result:\n%s',
info['call'],
info['failed'] and 'failed' or 'succeeded',
info['thread'], info['result'])
Added: zc.async/branches/dev/src/zc/async/multidb_dispatcher_policy.zcml
===================================================================
--- zc.async/branches/dev/src/zc/async/multidb_dispatcher_policy.zcml (rev 0)
+++ zc.async/branches/dev/src/zc/async/multidb_dispatcher_policy.zcml 2008-04-08 03:25:09 UTC (rev 85155)
@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<configure xmlns="http://namespaces.zope.org/zope">
+ <include file="dispatcher.zcml" />
+ <subscriber handler=".subscribers.multidb_queue_installer" />
+ <subscriber handler=".subscribers.threaded_dispatcher_installer" />
+ <subscriber handler=".subscribers.agent_installer" />
+ <adapter factory="zc.async.queue.getDefaultQueue" />
+</configure>
Modified: zc.async/branches/dev/src/zc/async/queue.py
===================================================================
--- zc.async/branches/dev/src/zc/async/queue.py 2008-04-07 16:01:47 UTC (rev 85154)
+++ zc.async/branches/dev/src/zc/async/queue.py 2008-04-08 03:25:09 UTC (rev 85155)
@@ -217,7 +217,7 @@
zope.interface.implements(zc.async.interfaces.IQueue)
def __init__(self):
- self._queue = zc.queue.CompositePersistentQueue()
+ self._queue = zc.queue.CompositeQueue()
self._held = BTrees.OOBTree.OOBTree()
self.quotas = Quotas()
self.quotas.__parent__ = self
Modified: zc.async/branches/dev/src/zc/async/tests.py
===================================================================
--- zc.async/branches/dev/src/zc/async/tests.py 2008-04-07 16:01:47 UTC (rev 85154)
+++ zc.async/branches/dev/src/zc/async/tests.py 2008-04-08 03:25:09 UTC (rev 85155)
@@ -1,7 +1,7 @@
import os
import unittest
-from zope.testing import doctest, module
+from zope.testing import doctest, module, loggingsupport
import zope.component
import zope.component.testing
import zope.component.eventtesting
@@ -25,10 +25,16 @@
zope.component.testing.setUp(test)
module.setUp(test, 'zc.async.doctest_test')
zope.component.eventtesting.setUp(test)
+ test.globs['event_logs'] = loggingsupport.InstalledHandler(
+ 'zc.async.events')
+ test.globs['trace_logs'] = loggingsupport.InstalledHandler(
+ 'zc.async.trace')
def modTearDown(test):
import transaction
transaction.abort()
+ import zc.async.dispatcher
+ zc.async.dispatcher.clear()
uuidTearDown(test)
zc.async.testing.tearDownDatetime()
module.tearDown(test)
@@ -43,6 +49,9 @@
test.globs['async_db'].close()
test.globs['async_storage'].close()
test.globs['async_storage'].cleanup()
+ for logs in (test.globs['event_logs'], test.globs['trace_logs']):
+ logs.clear()
+ logs.uninstall()
def test_instanceuuid():
"""This module provides access to a UUID that is intended to uniquely
Modified: zc.async/branches/dev/src/zc/async/z3tests.py
===================================================================
--- zc.async/branches/dev/src/zc/async/z3tests.py 2008-04-07 16:01:47 UTC (rev 85154)
+++ zc.async/branches/dev/src/zc/async/z3tests.py 2008-04-08 03:25:09 UTC (rev 85155)
@@ -1,6 +1,7 @@
import os
import unittest
from zope.testing import doctest, module
+import zope.component.testing
import zc.async.tests
@@ -19,6 +20,11 @@
'monitor.txt',
setUp=setUp, tearDown=zc.async.tests.modTearDown,
optionflags=doctest.INTERPRET_FOOTNOTES),
+ doctest.DocFileSuite(
+ 'README_3.txt',
+ setUp=zope.component.testing.setUp,
+ tearDown=zope.component.testing.tearDown,
+ optionflags=doctest.INTERPRET_FOOTNOTES),
))
More information about the Checkins
mailing list