[Checkins] SVN: z3c.saconfig/trunk/s Make engine lookup work,
make local engines work.
Martijn Faassen
faassen at infrae.com
Thu Jun 19 17:47:01 EDT 2008
Log message for revision 87571:
Make engine lookup work, make local engines work.
Changed:
U z3c.saconfig/trunk/setup.py
U z3c.saconfig/trunk/src/z3c/saconfig/README.txt
U z3c.saconfig/trunk/src/z3c/saconfig/__init__.py
U z3c.saconfig/trunk/src/z3c/saconfig/interfaces.py
U z3c.saconfig/trunk/src/z3c/saconfig/scopedsession.py
U z3c.saconfig/trunk/src/z3c/saconfig/tests.py
U z3c.saconfig/trunk/src/z3c/saconfig/utility.py
-=-
Modified: z3c.saconfig/trunk/setup.py
===================================================================
--- z3c.saconfig/trunk/setup.py 2008-06-19 21:43:38 UTC (rev 87570)
+++ z3c.saconfig/trunk/setup.py 2008-06-19 21:46:45 UTC (rev 87571)
@@ -33,6 +33,7 @@
'zope.sqlalchemy',
'zope.interface',
'zope.component',
+ 'zope.hookable',
],
entry_points="""
# -*- Entry points: -*-
Modified: z3c.saconfig/trunk/src/z3c/saconfig/README.txt
===================================================================
--- z3c.saconfig/trunk/src/z3c/saconfig/README.txt 2008-06-19 21:43:38 UTC (rev 87570)
+++ z3c.saconfig/trunk/src/z3c/saconfig/README.txt 2008-06-19 21:46:45 UTC (rev 87571)
@@ -15,7 +15,7 @@
The simplest way to set up SQLAlchemy for Zope is to have a single
thread-scoped session that's global to your entire Zope
instance. Multiple applications will all share this session. The
-engine is set up globally, in code.
+engine is set up with a global utility.
We use the SQLAlchemy ``sqlalchemy.ext.declarative`` extension to
define some tables and classes::
@@ -36,18 +36,41 @@
... email = Column('email', String(50))
... user_id = Column('user_id', Integer, ForeignKey('test_users.id'))
-We now set up an engine globally with our test DSN::
+So far this doesn't differ from the ``zope.sqlalchemy`` example. We
+now arrive at the first difference. Instead of making the engine
+directly, we can set up the engine factory as a (global) utility. This
+utility makes sure an engine is created and cached for us.
- >>> engine = create_engine(TEST_DSN, convert_unicode=True)
+ >>> from z3c.saconfig import EngineFactory
+ >>> engine_factory = EngineFactory(TEST_DSN)
-And we create the tables in our test database::
+You can pass the parameters you'd normally pass to
+``sqlalchemy.create_engine`` to ``EngineFactory``. Note that
+``z3c.saconfig`` assumes ``convert_unicode`` to be ``True`` by
+default.
+We now register the engine factory as a global utility using
+``zope.component``. Normally you'd use either ZCML or Grok to do this
+confirmation, but we'll do it manually here::::
+
+ >>> from zope import component
+ >>> from z3c.saconfig.interfaces import IEngineFactory
+ >>> component.provideUtility(engine_factory, provides=IEngineFactory)
+
+Note that setting up an engine factory is not actually necessary in
+the globally scoped use case. You could also just create the engine as
+a global and pass it as ``bind`` when you create the
+``GloballyScopedSession`` later.
+
+Let's look up the engine by calling the factory and create the tables
+in our test database::
+
+ >>> engine = engine_factory()
>>> Base.metadata.create_all(engine)
-So far this example doesn't differ any from the way
-``zope.sqlalchemy`` operates. The difference is in how we set up the
-session and use it. We'll use the ``GloballyScopedSession`` utility
-to implement our session creation::
+Now as for the second difference from ``zope.sqlalchemy``: how the
+session is set up and used. We'll use the ``GloballyScopedSession``
+utility to implement our session creation::
>>> from z3c.saconfig import GloballyScopedSession
@@ -55,24 +78,24 @@
you'd normally give to ``sqlalchemy.orm.create_session``, or
``sqlalchemy.orm.sessionmaker``::
- >>> utility = GloballyScopedSession(
- ... bind=engine,
- ... twophase=TEST_TWOPHASE)
+ >>> utility = GloballyScopedSession(twophase=TEST_TWOPHASE)
-``GloballyScopedSession`` automatically sets up the ``autocommit``,
-``autoflush`` and ``extension`` parameters to be the right ones for
-Zope integration, so normally you wouldn't need to supply these, but
-you could pass in your own if you do need it.
+``GlobalScopedSession`` looks up the engine using ``IEngineFactory``
+if you don't supply your own ``bind``
+argument. ``GloballyScopedSession`` also automatically sets up the
+``autocommit``, ``autoflush`` and ``extension`` parameters to be the
+right ones for Zope integration, so normally you wouldn't need to
+supply these, but you could pass in your own if you do need it.
-We now register this as an ``IScopedSession`` utility with
-``zope.component``. Normally you'd use either ZCML or Grok to do this
-confirmation, but we'll do it manually here::
+We now register this as an ``IScopedSession`` utility::
- >>> from zope import component
>>> from z3c.saconfig.interfaces import IScopedSession
>>> component.provideUtility(utility, provides=IScopedSession)
-We can now use the ``Session`` object create a session which
+We are done with configuration now. As you have seen it involves
+setting up two utilities, ``IEngineFactory`` and ``IScopedSession``,
+where only the latter is really needed in this globally shared session
+use case. We can now use the ``Session`` object create a session which
will behave according to the utility we provided::
>>> from z3c.saconfig import Session
@@ -94,6 +117,132 @@
>>> bob.addresses
[]
+SiteScopedSession
+=================
+
+In the example above we have set up SQLAlchemy with Zope using
+utilities, but it did not gain us very much, except that you can just
+use ``z3c.saconfig.Session`` to get the correct session.
+
+Now we'll see how we can set up different engines per site by
+registering the engine factory as a local utility for each one.
+
+In order to make this work, we'll set up ``SiteScopedSession`` instead
+of ``GloballyScopedSession``. We need to subclass
+``SiteScopedSession`` first because we need to implement its
+``siteScopeFunc`` method, which should return a unique ID per site
+(such as a path retrieved by ``zope.traversing.api.getPath``). We need
+to implement it here, as ``z3c.saconfig`` leaves this policy up to the
+application or a higher-level framework::
+
+ >>> from z3c.saconfig import SiteScopedSession
+ >>> class OurSiteScopedSession(SiteScopedSession):
+ ... def siteScopeFunc(self):
+ ... return getSite().id # the dummy site has a unique id
+ >>> utility = OurSiteScopedSession()
+ >>> component.provideUtility(utility, provides=IScopedSession)
+
+We want to register two engine factories, each in a different site::
+
+ >>> engine_factory1 = EngineFactory(TEST_DSN1)
+ >>> engine_factory2 = EngineFactory(TEST_DSN2)
+
+We need to set up the database in both new engines::
+
+ >>> Base.metadata.create_all(engine_factory1())
+ >>> Base.metadata.create_all(engine_factory2())
+
+Let's now create two sites, each of which will be connected to another
+engine::
+
+ >>> site1 = DummySite(id=1)
+ >>> site2 = DummySite(id=2)
+
+We set the local engine factories for each site:
+
+ >>> sm1 = site1.getSiteManager()
+ >>> sm1.registerUtility(engine_factory1, provided=IEngineFactory)
+ >>> sm2 = site2.getSiteManager()
+ >>> sm2.registerUtility(engine_factory2, provided=IEngineFactory)
+
+Just so we don't accidentally get it, we'll disable our global engine factory::
+
+ >>> component.provideUtility(None, provides=IEngineFactory)
+
+When we set the site to ``site1``, a lookup of ``IEngineFactory`` gets
+us engine factory 1::
+
+ >>> setSite(site1)
+ >>> component.getUtility(IEngineFactory) is engine_factory1
+ True
+
+And when we set it to ``site2``, we'll get engine factory 2::
+
+ >>> setSite(site2)
+ >>> component.getUtility(IEngineFactory) is engine_factory2
+ True
+
+We can look up our global utility even if we're in a site::
+
+ >>> component.getUtility(IScopedSession) is utility
+ True
+
+Phew. That was a lot of set up, but basically this is actually just
+straightforward utility setup code; you should use the APIs or Grok's
+``grok.local_utility`` directive to set up local utilities. Now all
+that is out of the way, we can create a session for ``site1``::
+
+ >>> setSite(site1)
+ >>> session = Session()
+
+The database is still empty::
+
+ >>> session.query(User).all()
+ []
+
+We'll add something to this database now::
+
+ >>> session.save(User(name='bob'))
+ >>> transaction.commit()
+
+``bob`` is now there::
+
+ >>> session = Session()
+ >>> session.query(User).all()[0].name
+ u'bob'
+
+Now we'll switch to ``site2``::
+
+ >>> setSite(site2)
+
+If we create a new session now, we should now be working with a
+different database, which should still be empty::
+
+ >>> session = Session()
+ >>> session.query(User).all()
+ []
+
+We'll add ``fred`` to this database::
+
+ >>> session.save(User(name='fred'))
+ >>> transaction.commit()
+
+Now ``fred`` is indeed there::
+
+ >>> session = Session()
+ >>> session.query(User).all()[0].name
+ u'fred'
+
+And ``bob`` is still in ``site1``::
+
+ >>> setSite(site1)
+ >>> session = Session()
+ >>> users = session.query(User).all()
+ >>> len(users)
+ 1
+ >>> users[0].name
+ u'bob'
+
Running the tests
=================
Modified: z3c.saconfig/trunk/src/z3c/saconfig/__init__.py
===================================================================
--- z3c.saconfig/trunk/src/z3c/saconfig/__init__.py 2008-06-19 21:43:38 UTC (rev 87570)
+++ z3c.saconfig/trunk/src/z3c/saconfig/__init__.py 2008-06-19 21:46:45 UTC (rev 87571)
@@ -1,2 +1,3 @@
from z3c.saconfig.scopedsession import Session
-from z3c.saconfig.utility import GloballyScopedSession
+from z3c.saconfig.utility import (
+ GloballyScopedSession, SiteScopedSession, EngineFactory)
Modified: z3c.saconfig/trunk/src/z3c/saconfig/interfaces.py
===================================================================
--- z3c.saconfig/trunk/src/z3c/saconfig/interfaces.py 2008-06-19 21:43:38 UTC (rev 87570)
+++ z3c.saconfig/trunk/src/z3c/saconfig/interfaces.py 2008-06-19 21:46:45 UTC (rev 87571)
@@ -8,14 +8,14 @@
to transparently use a different engine and session configuration per
database.
"""
- def session_factory():
+ def sessionFactory():
"""Create a SQLAlchemy session.
Typically you'd use sqlalchemy.orm.create_session to create
the session here.
"""
- def scopefunc():
+ def scopeFunc():
"""Determine the scope of the session.
This can be used to scope the session per thread, per Zope 3 site,
@@ -23,4 +23,48 @@
like a thread id, or a tuple with thread id and application id.
"""
+class ISiteScopedSession(IScopedSession):
+ """A utility that makes sessions be scoped by site.
+ """
+ def siteScopeFunc():
+ """Returns a unique id per site.
+ """
+class IEngineFactory(Interface):
+ """A utility that represents an SQLAlchemy engine.
+
+ If the engine isn't created yet, it will create it. Otherwise the
+ engine will be cached.
+
+ When an engine property is changed, the engine will be recreated
+ dynamically.
+ """
+
+ def __call__():
+ """Get the engine.
+
+ This creates the engine if this factory was not used before,
+ otherwise returns a cached version.
+ """
+
+ def configuration():
+ """Returns the engine configuration in the form of an args, kw tuple.
+
+ Return the parameters used to create an engine as a tuple with
+ an args list and a kw dictionary.
+ """
+
+ def reset():
+ """Reset the cached engine (if any).
+
+ This causes the engine to be recreated on next use.
+ """
+
+ def getCached():
+ """Return the cached engine.
+ """
+
+ def cache(engine):
+ """Cache the engine.
+ """
+
Modified: z3c.saconfig/trunk/src/z3c/saconfig/scopedsession.py
===================================================================
--- z3c.saconfig/trunk/src/z3c/saconfig/scopedsession.py 2008-06-19 21:43:38 UTC (rev 87570)
+++ z3c.saconfig/trunk/src/z3c/saconfig/scopedsession.py 2008-06-19 21:46:45 UTC (rev 87571)
@@ -9,7 +9,7 @@
It delegates to a IScopedSession utility.
"""
utility = component.getUtility(IScopedSession)
- return utility.session_factory()
+ return utility.sessionFactory()
def scopefunc():
"""This is used by scoped session to distinguish between sessions.
@@ -17,7 +17,7 @@
It delegates to a IScopedSession utility.
"""
utility = component.getUtility(IScopedSession)
- return utility.scopefunc()
+ return utility.scopeFunc()
# this is framework central configuration. Use a IScopedSession utility
# to define behavior.
Modified: z3c.saconfig/trunk/src/z3c/saconfig/tests.py
===================================================================
--- z3c.saconfig/trunk/src/z3c/saconfig/tests.py 2008-06-19 21:43:38 UTC (rev 87570)
+++ z3c.saconfig/trunk/src/z3c/saconfig/tests.py 2008-06-19 21:46:45 UTC (rev 87571)
@@ -3,7 +3,10 @@
#
# export TEST_DSN=postgres://plone:plone@localhost/test
# export TEST_DSN=mssql://plone:plone@/test?dsn=mydsn
-#
+
+# Since the test exercise what happens with two different DSNs
+# locally, you need to also set up a different TEST_DSN2.
+
# To test in twophase commit mode export TEST_TWOPHASE=True
#
# NOTE: The sqlite that ships with Mac OS X 10.4 is buggy.
@@ -15,10 +18,22 @@
import os
from zope.testing import cleanup
+from zope.testing.cleanup import addCleanUp
+from zope import component
+from zope.component import registry
+
TEST_TWOPHASE = bool(os.environ.get('TEST_TWOPHASE'))
TEST_DSN = os.environ.get('TEST_DSN', 'sqlite:///:memory:')
+TEST_DSN1 = TEST_DSN
+# this can reuse TEST_DSN1 in the default case, as we can open another
+# in-memory database. You can't do this for other databases however.
+TEST_DSN2 = os.environ.get('TEST_DSN', TEST_DSN1)
+def setUpReadMe(test):
+ # set up special local component architecture
+ setHooks()
+
def tearDownReadMe(test):
# clean up Zope
cleanup.cleanUp()
@@ -28,13 +43,71 @@
engine = test.globs['engine']
Base.metadata.drop_all(engine)
+# a very simple implementation of setSite and getSite so we don't have
+# to rely on zope.app.component just for our tests
+_site = None
+
+class DummySite(object):
+ def __init__(self, id):
+ self.id = id
+ self._sm = SiteManager()
+
+ def getSiteManager(self):
+ return self._sm
+
+class SiteManager(registry.Components):
+ def __init__(self):
+ super(SiteManager, self).__init__()
+ self.__bases__ = (component.getGlobalSiteManager(),)
+
+def setSite(site=None):
+ global _site
+ _site = site
+
+def getSite():
+ return _site
+
+def adapter_hook(interface, object, name='', default=None):
+ try:
+ return getSiteManager().adapters.adapter_hook(
+ interface, object, name, default)
+ except component.interfaces.ComponentLookupError:
+ return default
+
+def getSiteManager(context=None):
+ if _site is not None:
+ return _site.getSiteManager()
+ return component.getGlobalSiteManager()
+
+def setHooks():
+ component.adapter_hook.sethook(adapter_hook)
+ component.getSiteManager.sethook(getSiteManager)
+
+def resetHooks():
+ component.adapter_hook.reset()
+ component.getSiteManager.reset()
+
+# make sure hooks get cleaned up after tests are run
+addCleanUp(resetHooks)
+
def test_suite():
optionflags = doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS
-
+ globs = {
+ 'DummySite': DummySite,
+ 'setSite': setSite,
+ 'getSite': getSite,
+ 'TEST_DSN': TEST_DSN,
+ 'TEST_DSN1': TEST_DSN1,
+ 'TEST_DSN2': TEST_DSN2,
+ 'TEST_TWOPHASE': TEST_TWOPHASE,
+ }
+
suite = unittest.TestSuite()
+
suite.addTest(doctest.DocFileSuite(
'README.txt',
optionflags=optionflags,
+ setUp=setUpReadMe,
tearDown=tearDownReadMe,
- globs={'TEST_DSN': TEST_DSN, 'TEST_TWOPHASE': TEST_TWOPHASE}))
+ globs=globs))
return suite
Modified: z3c.saconfig/trunk/src/z3c/saconfig/utility.py
===================================================================
--- z3c.saconfig/trunk/src/z3c/saconfig/utility.py 2008-06-19 21:43:38 UTC (rev 87570)
+++ z3c.saconfig/trunk/src/z3c/saconfig/utility.py 2008-06-19 21:46:45 UTC (rev 87571)
@@ -3,11 +3,14 @@
"""
import thread
+import sqlalchemy
+
from zope.interface import implements
-import sqlalchemy
+from zope import component
from zope.sqlalchemy import ZopeTransactionExtension
-from z3c.saconfig.interfaces import IScopedSession
+from z3c.saconfig.interfaces import (IScopedSession, ISiteScopedSession,
+ IEngineFactory)
class GloballyScopedSession(object):
"""A globally scoped session.
@@ -36,16 +39,112 @@
Normally you wouldn't pass these in, but if you have the need
to override them, you could.
"""
- if 'autocommit' not in kw:
- kw['autocommit'] = False
- if 'autoflush' not in kw:
- kw['autoflush'] = True
- if 'extension' not in kw:
- kw['extension'] = ZopeTransactionExtension()
- self.kw = kw
+ self.kw = _zope_session_defaults(kw)
- def session_factory(self):
- return sqlalchemy.orm.create_session(**self.kw)
+ def sessionFactory(self):
+ kw = self.kw.copy()
+ if 'bind' not in kw:
+ # look up the engine using IEngineFactory if needed
+ engine_factory = component.getUtility(IEngineFactory)
+ kw['bind'] = engine_factory()
+ return sqlalchemy.orm.create_session(**kw)
- def scopefunc(self):
+ def scopeFunc(self):
return thread.get_ident()
+
+def _zope_session_defaults(kw):
+ """Adjust keyword parameters with proper defaults for Zope.
+ """
+ kw = kw.copy()
+ if 'autocommit' not in kw:
+ kw['autocommit'] = False
+ if 'autoflush' not in kw:
+ kw['autoflush'] = True
+ if 'extension' not in kw:
+ kw['extension'] = ZopeTransactionExtension()
+ return kw
+
+class SiteScopedSession(object):
+ """A session that is scoped per site.
+
+ Even though this makes the sessions scoped per site,
+ the utility can be registered globally to make this work.
+
+ Creation arguments as for GloballyScopedSession, except that no ``bind``
+ parameter should be passed. This means it is possible to create
+ a SiteScopedSession utility without passing parameters to its constructor.
+ """
+ implements(ISiteScopedSession)
+
+ def __init__(self, **kw):
+ assert 'bind' not in kw
+ self.kw = _zope_session_defaults(kw)
+
+ def sessionFactory(self):
+ engine_factory = component.getUtility(IEngineFactory)
+ kw = self.kw.copy()
+ kw['bind'] = engine_factory()
+ return sqlalchemy.orm.create_session(**kw)
+
+ def scopeFunc(self):
+ return (thread.get_ident(), self.siteScopeFunc())
+
+ def siteScopeFunc(self):
+ raise NotImplementedError
+
+class EngineFactory(object):
+ """An engine factory.
+
+ If you need engine connection parameters to be different per site,
+ EngineFactory should be registered as a local utility in that
+ site.
+
+ convert_unicode is True by default.
+
+ If you want this utility to be persistent, you should subclass it
+ and mixin Persistent. You could then manage the parameters
+ differently than is done in this __init__, for instance as
+ attributes, which is nicer if you are using Persistent (or Zope 3
+ schema). In this case you need to override the configuration method.
+ """
+ implements(IEngineFactory)
+
+ def __init__(self, *args, **kw):
+ if 'convert_unicode' not in kw:
+ kw['convert_unicode'] = True
+ self._args = args
+ self._kw = kw
+
+ def __call__(self):
+ engine = self.getCached()
+ if engine is not None:
+ return engine
+ # no engine yet, so create a new one
+ args, kw = self.configuration()
+ engine = sqlalchemy.create_engine(*args, **kw)
+ self.cache(engine)
+ return engine
+
+ def configuration(self):
+ """Returns engine parameters.
+
+ This can be overridden in a subclass to retrieve the parameters
+ from some other place.
+ """
+ return self._args, self._kw
+
+ def reset(self):
+ engine = self.getCached()
+ if engine is None:
+ return
+ # XXX is disposing the right thing to do?
+ engine.dispose()
+ self.cache(None)
+
+ # XXX what happens if EngineFactory were to be evicted from the ZODB
+ # cache?
+ def getCached(self):
+ return getattr(self, '_v_engine', None)
+
+ def cache(self, engine):
+ self._v_engine = engine
More information about the Checkins
mailing list