[Checkins] SVN: zc.virtualstorage/trunk/ initial checkin
Gary Poster
gary at zope.com
Sun Dec 2 06:22:39 EST 2007
Log message for revision 82076:
initial checkin
Changed:
_U zc.virtualstorage/trunk/
A zc.virtualstorage/trunk/CHANGES.txt
A zc.virtualstorage/trunk/README.txt
A zc.virtualstorage/trunk/ZopePublicLicense.txt
A zc.virtualstorage/trunk/bootstrap.py
A zc.virtualstorage/trunk/buildout.cfg
A zc.virtualstorage/trunk/setup.py
A zc.virtualstorage/trunk/src/
A zc.virtualstorage/trunk/src/zc/
A zc.virtualstorage/trunk/src/zc/__init__.py
A zc.virtualstorage/trunk/src/zc/virtualstorage/
A zc.virtualstorage/trunk/src/zc/virtualstorage/CHANGES.txt
A zc.virtualstorage/trunk/src/zc/virtualstorage/README.txt
A zc.virtualstorage/trunk/src/zc/virtualstorage/__init__.py
A zc.virtualstorage/trunk/src/zc/virtualstorage/base.py
A zc.virtualstorage/trunk/src/zc/virtualstorage/component.xml
A zc.virtualstorage/trunk/src/zc/virtualstorage/config.py
A zc.virtualstorage/trunk/src/zc/virtualstorage/tests.py
-=-
Property changes on: zc.virtualstorage/trunk
___________________________________________________________________
Name: svn:ignore
+ develop-eggs
bin
parts
.installed.cfg
build
dist
eggs
Name: svn:externals
+ zodb svn+ssh://svn.zope.org/repos/main/ZODB/trunk
Added: zc.virtualstorage/trunk/CHANGES.txt
===================================================================
--- zc.virtualstorage/trunk/CHANGES.txt (rev 0)
+++ zc.virtualstorage/trunk/CHANGES.txt 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1 @@
+Please see CHANGES.txt in src/zc/virtualstorage.
Property changes on: zc.virtualstorage/trunk/CHANGES.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.virtualstorage/trunk/README.txt
===================================================================
--- zc.virtualstorage/trunk/README.txt (rev 0)
+++ zc.virtualstorage/trunk/README.txt 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1 @@
+Create virtual ZODB storages, usable with some revision control semantics.
Property changes on: zc.virtualstorage/trunk/README.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.virtualstorage/trunk/ZopePublicLicense.txt
===================================================================
--- zc.virtualstorage/trunk/ZopePublicLicense.txt (rev 0)
+++ zc.virtualstorage/trunk/ZopePublicLicense.txt 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,54 @@
+Zope Public License (ZPL) Version 2.1
+-------------------------------------
+
+A copyright notice accompanies this license document that
+identifies the copyright holders.
+
+This license has been certified as open source. It has also
+been designated as GPL compatible by the Free Software
+Foundation (FSF).
+
+Redistribution and use in source and binary forms, with or
+without modification, are permitted provided that the
+following conditions are met:
+
+1. Redistributions in source code must retain the
+ accompanying copyright notice, this list of conditions,
+ and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the accompanying
+ copyright notice, this list of conditions, and the
+ following disclaimer in the documentation and/or other
+ materials provided with the distribution.
+
+3. Names of the copyright holders must not be used to
+ endorse or promote products derived from this software
+ without prior written permission from the copyright
+ holders.
+
+4. The right to distribute this software or to use it for
+ any purpose does not give you the right to use
+ Servicemarks (sm) or Trademarks (tm) of the copyright
+ holders. Use of them is covered by separate agreement
+ with the copyright holders.
+
+5. If any files are modified, you must cause the modified
+ files to carry prominent notices stating that you changed
+ the files and the date of any change.
+
+Disclaimer
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS''
+ AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT
+ NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
+ AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN
+ NO EVENT SHALL THE COPYRIGHT HOLDERS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+ OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+ DAMAGE.
Property changes on: zc.virtualstorage/trunk/ZopePublicLicense.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.virtualstorage/trunk/bootstrap.py
===================================================================
--- zc.virtualstorage/trunk/bootstrap.py (rev 0)
+++ zc.virtualstorage/trunk/bootstrap.py 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,52 @@
+##############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Bootstrap a buildout-based project
+
+Simply run this script in a directory containing a buildout.cfg.
+The script accepts buildout command-line options, so you can
+use the -c option to specify an alternate configuration file.
+
+$Id: bootstrap.py 69908 2006-08-31 21:53:00Z jim $
+"""
+
+import os, shutil, sys, tempfile, urllib2
+
+tmpeggs = tempfile.mkdtemp()
+
+ez = {}
+exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
+ ).read() in ez
+ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
+
+import pkg_resources
+
+cmd = 'from setuptools.command.easy_install import main; main()'
+if sys.platform == 'win32':
+ cmd = '"%s"' % cmd # work around spawn lamosity on windows
+
+ws = pkg_resources.working_set
+assert os.spawnle(
+ os.P_WAIT, sys.executable, sys.executable,
+ '-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout',
+ dict(os.environ,
+ PYTHONPATH=
+ ws.find(pkg_resources.Requirement.parse('setuptools')).location
+ ),
+ ) == 0
+
+ws.add_entry(tmpeggs)
+ws.require('zc.buildout')
+import zc.buildout.buildout
+zc.buildout.buildout.main(sys.argv[1:] + ['bootstrap'])
+shutil.rmtree(tmpeggs)
Added: zc.virtualstorage/trunk/buildout.cfg
===================================================================
--- zc.virtualstorage/trunk/buildout.cfg (rev 0)
+++ zc.virtualstorage/trunk/buildout.cfg 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,15 @@
+[buildout]
+develop = . zodb zodb/transaction
+parts = test py
+
+find-links = http://download.zope.org/distribution/
+
+[test]
+recipe = zc.recipe.testrunner
+eggs = zc.virtualstorage
+defaults = "--tests-pattern [fn]?tests --exit-with-status".split()
+
+[py]
+recipe = zc.recipe.egg
+eggs = zc.virtualstorage
+interpreter = py
Property changes on: zc.virtualstorage/trunk/buildout.cfg
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.virtualstorage/trunk/setup.py
===================================================================
--- zc.virtualstorage/trunk/setup.py (rev 0)
+++ zc.virtualstorage/trunk/setup.py 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,29 @@
+from setuptools import setup, find_packages
+
+setup(
+ name="zc.virtualstorage",
+ version="0.1",
+ packages=find_packages('src'),
+ include_package_data=True,
+ package_dir= {'':'src'},
+
+ namespace_packages=['zc'],
+
+ zip_safe=False,
+ author='Zope Project',
+ author_email='zope-dev at zope.org',
+ description=open("README.txt").read(),
+ long_description=(
+ open('src/zc/virtualstorage/CHANGES.txt').read() +
+ '\n========\nOverview\n========\n\n' +
+ open("src/zc/virtualstorage/README.txt").read()),
+ license='ZPL 2.1',
+ keywords="zope zope3",
+ install_requires=[
+ 'ZODB3 >= 3.9dev',
+ 'zope.interface',
+ 'setuptools',
+
+ 'zope.testing',
+ ],
+ )
Added: zc.virtualstorage/trunk/src/zc/__init__.py
===================================================================
--- zc.virtualstorage/trunk/src/zc/__init__.py (rev 0)
+++ zc.virtualstorage/trunk/src/zc/__init__.py 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,7 @@
+# this is a namespace package
+try:
+ import pkg_resources
+ pkg_resources.declare_namespace(__name__)
+except ImportError:
+ import pkgutil
+ __path__ = pkgutil.extend_path(__path__, __name__)
Added: zc.virtualstorage/trunk/src/zc/virtualstorage/CHANGES.txt
===================================================================
--- zc.virtualstorage/trunk/src/zc/virtualstorage/CHANGES.txt (rev 0)
+++ zc.virtualstorage/trunk/src/zc/virtualstorage/CHANGES.txt 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,10 @@
+=======
+Changes
+=======
+
+0.1
+===
+
+(dev version targeting ZODB 3.9)
+
+Initial release
\ No newline at end of file
Property changes on: zc.virtualstorage/trunk/src/zc/virtualstorage/CHANGES.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.virtualstorage/trunk/src/zc/virtualstorage/README.txt
===================================================================
--- zc.virtualstorage/trunk/src/zc/virtualstorage/README.txt (rev 0)
+++ zc.virtualstorage/trunk/src/zc/virtualstorage/README.txt 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,609 @@
+=================
+zc.virtualstorage
+=================
+
+------------
+Introduction
+------------
+
+zc.virtualstorage allows largely transparent versioning of ZODB object
+collections. This can be used to provide staging semantics for a collection of
+objects such as dev-staged-prod for a retail site hierarchy or
+sandboxes-branches-trunk for a collection of interconnected configuration
+objects [#zc.vault]_.
+
+Basic Usage
+===========
+
+To use zc.virtualstorage, you must wrap a real storage (any type, in theory;
+the developer will use and directly support FileStorage and ZEO client
+storage) with a custom DB subclass.
+
+ >>> import ZODB.FileStorage
+ >>> storage = ZODB.FileStorage.FileStorage(
+ ... 'HistoricalConnectionTests.fs', create=True)
+ >>> import zc.virtualstorage.base
+ >>> db = zc.virtualstorage.base.DB(storage)
+
+A connection from this DB instance returns a subclass of the ZODB Connection
+called a Coordinator. The coordinator does everything that a normal connection
+does and can be used for normal database interactions.
+
+ >>> coordinator = db.open()
+ >>> isinstance(coordinator, zc.virtualstorage.base.Coordinator)
+ True
+
+The Coordinator also coordinates the two-phase commit process for itself and
+any subsidiary connections to virtual storages.
+
+You can add a virtual storage anywhere in the database. Without using it to
+open a virtual connection, it's just another persistent object with nothing
+particularly unusual. We'll add one in the root.
+
+ >>> zero = zc.virtualstorage.base.VirtualStorage()
+ >>> coordinator.root()['zero'] = zero
+
+To actually connect to a virtual storage, the storage must have a _p_jar and
+a _p_oid to the coordinator (the main connection). There are three ways of
+doing that: committing the transaction, explicitly ``add``ing the object to the
+coordinator connection, or providing an adapter to ZODB.interfaces.IConnection
+that does the job. We'll ``add`` for now, and then ask the db to get a virtual
+connection.
+
+ >>> coordinator.add(zero)
+ >>> conn0 = db.getVirtualConnection(zero)
+
+You now have an open virtual connection to the virtual storage. As long as this
+virtual connection is open, you can call ``getVirtualConnection`` again with
+the same virtual storage and get the same connection [#get_is_recallable]_.
+
+Initially the virtual storage doesn't have a root: you need to ``add`` an object to a
+connection to register a root.
+
+ >>> conn0.root()
+ Traceback (most recent call last):
+ ...
+ POSKeyError: 0x00
+ >>> import persistent.mapping
+ >>> root = persistent.mapping.PersistentMapping()
+ >>> conn0.add(root)
+ >>> conn0.root() is root
+ True
+
+We can modify the root as desired.
+
+ >>> conn0.root()['answer'] = 17
+ >>> conn0.root()
+ {'answer': 17}
+ >>> conn0.root()['nested'] = persistent.mapping.PersistentMapping()
+ >>> conn0.root()['nested']['foo'] = 'bar'
+
+Now you can simply commit the transaction, and close the main connection if
+needed or desired.
+
+ >>> import transaction
+ >>> transaction.commit()
+ >>> coordinator.close()
+
+Additional connections get the same data, with invalidations working even on
+closed connections. Virtual connections to the same virtual storage are
+reused.
+
+ >>> coordinator = db.open()
+ >>> transaction2 = transaction.TransactionManager()
+ >>> coordinatorB = db.open(transaction_manager=transaction2)
+
+ >>> conn0B = db.getVirtualConnection(coordinatorB.root()['zero'])
+ >>> sorted(conn0B.root().keys())
+ ['answer', 'nested']
+ >>> coordinatorB.close()
+
+ >>> conn0 = db.getVirtualConnection(coordinator.root()['zero'])
+ >>> conn0 is conn0B # reusing the virtual connection
+ True
+ >>> conn0.root()['answer'] = 42
+ >>> transaction.commit()
+
+ >>> coordinatorB is db.open()
+ True
+ >>> conn0B = db.getVirtualConnection(coordinatorB.root()['zero'])
+ >>> conn0B.root()['answer']
+ 42
+ >>> coordinatorB.close()
+
+These virtual connections to virtual storages can work with blobs, savepoints,
+ZEO, historical connections, and undo, as seen in the advanced demonstrations
+at the end of the document.
+
+Storage Revisions
+=================
+
+This virtual storage is really just a curiosity until you start letting one
+virtual storage be based on another--allowing a frozen production version and
+an editable development version, for instance. To have one virtual storage
+based on another, the base must be readonly. In the base implementation,
+making a virtual storage is accomplished by the ``freeze`` method.
+
+ >>> zero = coordinator.root()['zero']
+ >>> zero.freeze()
+ >>> one = zc.virtualstorage.base.VirtualStorage(zero)
+ >>> coordinator.root()['one'] = one
+ >>> transaction.commit()
+
+Now the new storage looks like the old one.
+
+ >>> conn1 = db.getVirtualConnection(one)
+ >>> sorted(conn1.root().keys())
+ ['answer', 'nested']
+ >>> conn1.root()['answer']
+ 42
+ >>> conn1.root()['nested']
+ {'foo': 'bar'}
+
+We can mutate the new one, though, without affecting the old one.
+
+ >>> conn1.root()['blackjack'] = 21
+ >>> sorted(conn1.root().keys())
+ ['answer', 'blackjack', 'nested']
+
+ >>> conn0 = db.getVirtualConnection(zero)
+ >>> sorted(conn0.root().keys())
+ ['answer', 'nested']
+
+ >>> transaction.commit()
+
+The original storage is now frozen. If a connection to it is mutated,
+the transaction cannot be committed.
+
+ >>> conn0.root()['seven'] = 11
+ >>> transaction.commit()
+ Traceback (most recent call last):
+ ...
+ ReadOnlyError
+
+We'll abort our changes.
+
+ >>> transaction.abort()
+ >>> sorted(conn0.root().keys())
+ ['answer', 'nested']
+
+Caches and Memory
+=================
+
+When you open a connection to a virtual storage, this is a full-fledged
+connection as far as memory usage is concerned. The object cache is the biggest
+memory concern. It is very easy, then, to open many connections, if you have
+many storages, and quickly eat up unexpectedly large amounts of memory.
+
+The package offers three pairs of methods to try and protect memory usage:
+``getVirtualConnectionCacheSize`` and ``setVirtualConnectionCacheSize``;
+``getVirtualConnectionTimeout`` and ``setVirtualConnectionTimeout``; and
+``getVirtualConnectionPoolSize`` and ``setVirtualConnectionPoolSize``.
+
+* The cache size controls the target maximum number of objects in each virtual
+ connection's object cache.
+
+* The timeout controls the minimum number of seconds a closed virtual
+ connection is kept before it is discarded.
+
+* The count controls the maximum count of total virtual connections that are
+ kept when they close. The newest ones win.
+
+Of course, these have the same limitation as other standard ZODB object cache
+settings: the cache size is based on the number of objects rather than their
+actual total memory size. [#testcachecontrols]_.
+
+ZConfig
+=======
+
+To use virtual storage with ZConfig, simply ``%import zc.virtualstorage`` and
+use ``zodb_with_virtualstorage_support`` instead of ``zodb`` for the database
+you want to use with virtual storage. Here's the simplest example.
+
+ >>> import ZODB.config
+ >>> zconfig_db = ZODB.config.databaseFromString('''
+ ... %import zc.virtualstorage
+ ... <zodb_with_virtualstorage_support>
+ ... <mappingstorage/>
+ ... </zodb_with_virtualstorage_support>
+ ... ''')
+ >>> isinstance(zconfig_db, zc.virtualstorage.base.DB)
+ True
+ >>> zconfig_db.close()
+
+Within a multi-database process you can mix and match, using both
+the standard ``zodb`` and the custom ``zodb_with_virtualstorage_support``
+within your configuration file for different databases. Also, it's worth
+noting that you should be able to switch back to ``zodb`` if you decide to
+no longer use virtual storages: as mentioned before, from the perspective of
+the main connection, they are just persistent objects.
+
+You can provide all the same options to ``zodb_with_virtualstorage_support`` as
+to a standard ``zodb`` section. You can also specify the cache settings that we
+discussed in `Caches and Memory`_ above.
+
+ >>> zconfig_db = ZODB.config.databaseFromString('''
+ ... %import zc.virtualstorage
+ ... <zodb_with_virtualstorage_support>
+ ... virtual-cache-size 1492
+ ... virtual-timeout 12m
+ ... virtual-pool-size 7
+ ... <mappingstorage/>
+ ... </zodb_with_virtualstorage_support>
+ ... ''')
+ >>> zconfig_db.getVirtualConnectionCacheSize()
+ 1492
+ >>> zconfig_db.getVirtualConnectionTimeout() # 12 minutes * 60 seconds
+ 720
+ >>> zconfig_db.getVirtualConnectionPoolSize()
+ 7
+ >>> zconfig_db.close()
+
+Summary
+=======
+
+That's the basics of virtual storage: not much to it, which is the idea. To
+integrate with it, just use persistent objects as usual.
+
+The remainder of the document discusses how the virtual storage works and gives
+some advanced demonstrations of virtual storages working with standard ZODB
+features such as savepoints and blobs.
+
+------------
+How It Works
+------------
+
+While the virtual storage has to participate in the delicate dances of commit
+and cache invalidation, the basic concepts are simple. A virtual storage maps
+effective OIDs to real OIDs when necessary. This means that, for instance, when
+you ask for the object at the root OID of a virtual storage, the virtual
+storage maps the virtual root OID to a real OID, asks the real storage for that
+value, and then passes it back.
+
+We'll get an idea of this by looking at our two storages.
+
+The ``base`` attribute is a reference to the base storage.
+
+ >>> coordinator = db.open()
+ >>> one = coordinator.root()['one']
+ >>> zero = coordinator.root()['zero']
+ >>> one.base is zero
+ True
+ >>> zero.base is None
+ True
+
+If we look at the OIDs for the two persistent objects in each storage, they
+are the same.
+
+ >>> import ZODB.utils
+ >>> conn0 = db.getVirtualConnection(zero)
+ >>> conn1 = db.getVirtualConnection(one)
+ >>> conn1.root()._p_oid == ZODB.utils.z64
+ True
+ >>> conn0.root()._p_oid == conn1.root()._p_oid
+ True
+
+ >>> conn1.root()['nested']._p_oid == conn0.root()['nested']._p_oid
+ True
+
+However, the roots are different objects, while the 'nested' mappings share the
+same underlying object. The storages manage this with four data structures,
+which should generally be regarded as protected except during delicate jobs
+like generation scripts. ``local`` is a mapping from local overridden OID to
+real OID. Even for a storage without a base, this contains a mapping from the
+standard root OID, ZODB.utils.z64, to the real object used for this purpose.
+Storages with a base will also include any objects that mask ones in its base.
+
+ >>> list(zero.local.keys()) == [ZODB.utils.z64]
+ True
+ >>> list(one.local.keys()) == [ZODB.utils.z64]
+ True
+
+The ``reverse_local`` mapping is the mirror image of the ``local`` mapping,
+used for getLocalOID.
+
+ >>> list(zero.reverse_local.values()) == [ZODB.utils.z64]
+ True
+ >>> list(one.reverse_local.values()) == [ZODB.utils.z64]
+ True
+
+The ``new`` set is comprised of new objects within a storage that do not need
+mapping. Right now, zero.new contains the OID of the nested object, and
+one.new is empty.
+
+ >>> nested_oid = conn1.root()['nested']._p_oid
+ >>> set(zero.new) == set((nested_oid,))
+ True
+ >>> len(one.new)
+ 0
+
+Finally, the ``bucket`` attribute keeps a mapping of all local OIDs, new and
+local, to the actual objects used. The use case for this map is packing:
+we want the storage to have a direct, standard reference to all used objects
+so packing algorithms unaware of the zc.virtualstorage approach still keep
+the necessary objects around.
+
+ >>> dict((k, v._p_oid) for k, v in zero.bucket.items()) == {
+ ... ZODB.utils.z64: zero.local[ZODB.utils.z64],
+ ... nested_oid: nested_oid}
+ True
+ >>> dict((k, v._p_oid) for k, v in one.bucket.items()) == {
+ ... ZODB.utils.z64: one.local[ZODB.utils.z64]}
+ True
+ >>> (list(zero.reverse_local.keys()) == list(zero.local.values()) ==
+ ... [zero.bucket[ZODB.utils.z64]._p_oid])
+ True
+ >>> (list(one.reverse_local.keys()) == list(one.local.values()) ==
+ ... [one.bucket[ZODB.utils.z64]._p_oid])
+ True
+
+The ``bucket`` mapping may also be useful for jobs like generations scripts.
+While frozen storages will not allow changes to objects in the virtual
+connections, the collaborators will allow the commit, so you can use the
+bucket to iterate over all local objects in a frozen storage and do database
+generation work on them.
+
+Note that all four of these collections are typically only updated on a
+transaction commit. The only exception is the ``new`` collection, which will
+be modified when a virtual connection's ``add`` method is used successfuly.
+Even then, the modification will be discarded if the pertinent transaction is
+aborted.
+
+-----------------------
+Advanced Demonstrations
+-----------------------
+
+This section exercises built-in ZODB tricks to show that they can work with the
+virtual storage. As such, they are largely just confirmations and tests rather
+than new information.
+
+Savepoints
+==========
+
+Savepoints should work as they normally do. First we'll show a rollback.
+
+ >>> conn1.root()['abraham'] = 'abe'
+ >>> conn1.root()['nested']['emily'] = 'emma'
+ >>> sp = transaction.savepoint()
+ >>> conn1.root()['nested']['james'] = 'jim'
+ >>> conn1.root()['nested']['m'] = persistent.mapping.PersistentMapping()
+ >>> sp.rollback()
+
+Now we'll make some changes after a savepoint and commit.
+
+ >>> 'james' in conn1.root()['nested']
+ False
+ >>> 'm' in conn1.root()['nested']
+ False
+ >>> conn1.root()['abraham']
+ 'abe'
+ >>> conn1.root()['nested']['emily']
+ 'emma'
+ >>> conn1.root()['nested']['gerbrand'] = 'bran'
+ >>> transaction.commit()
+
+Blobs
+=====
+
+To show blobs, we need to use the storage with a blob proxy. We'll close our
+db, wrap our storage with a proxy, and make a new db.
+
+ >>> coordinator.close()
+ >>> from ZODB.blob import BlobStorage, Blob
+ >>> from tempfile import mkdtemp
+ >>> blob_dir = mkdtemp()
+ >>> blob_storage = BlobStorage(blob_dir, storage)
+ >>> db = zc.virtualstorage.base.DB(blob_storage)
+
+Now we'll put a blob in a virtual connection.
+
+ >>> blob = Blob()
+ >>> data = blob.open("w")
+ >>> data.write("I'm a happy Blob.")
+ >>> data.close()
+
+ >>> coordinator = db.open()
+ >>> conn1 = db.getVirtualConnection(coordinator.root()['one'])
+ >>> conn1.root()['nested']['blob'] = blob
+ >>> transaction.commit()
+
+The blob is available from other connections, as expected.
+
+ >>> coordinatorB = db.open(transaction_manager=transaction2)
+ >>> conn1B = db.getVirtualConnection(coordinatorB.root()['one'])
+ >>> conn1B.root()['nested']['blob'].open('r').read()
+ "I'm a happy Blob."
+
+ >>> conn1.close()
+ >>> coordinatorB.close()
+
+Basing one virtual storage on another also works with blobs: the original
+blob is untouched.
+
+ >>> coordinator.root()['one'].freeze()
+ >>> coordinator.root()['two'] = zc.virtualstorage.base.VirtualStorage(
+ ... coordinator.root()['one'])
+ >>> transaction.commit()
+
+ >>> conn2 = db.getVirtualConnection(coordinator.root()['two'])
+ >>> f = conn2.root()['nested']['blob'].open('w')
+ >>> f.write('I am an ecstatic Blob.')
+ >>> f.close()
+ >>> transaction.commit()
+
+ >>> coordinatorB = db.open(transaction_manager=transaction2)
+ >>> conn1B = db.getVirtualConnection(coordinatorB.root()['one'])
+ >>> conn1B.root()['nested']['blob'].open('r').read()
+ "I'm a happy Blob."
+
+ >>> conn2B = db.getVirtualConnection(coordinatorB.root()['two'])
+ >>> conn2B.root()['nested']['blob'].open('r').read()
+ 'I am an ecstatic Blob.'
+
+Conflict Errors
+===============
+
+Conflict errors work as usual.
+
+ >>> conn2.root()['nested']['yrag'] = 'gary'
+ >>> conn2B.root()['nested']['nyrak'] = 'karyn'
+ >>> transaction2.commit()
+ >>> transaction.commit() # doctest: +ELLIPSIS
+ Traceback (most recent call last):
+ ...
+ ConflictError: ...
+ >>> 'yrag' in conn2.root()['nested']
+ False
+ >>> conn2.root()['nested']['nyrak']
+ 'karyn'
+
+If you abort and retry, everything will work again, as usual.
+
+ >>> transaction.abort()
+ >>> conn2.root()['nested']['yrag'] = 'gary'
+ >>> transaction.commit()
+ >>> conn2.root()['nested']['yrag']
+ 'gary'
+ >>> conn2.root()['nested']['nyrak']
+ 'karyn'
+
+Historical Connections
+======================
+
+If you open up a main "coordinator" connection that is historical (using ``at``
+or ``before``) then the virtual storages and virtual connections are also tied
+to that time. We'll show this by opening up an historical connection when
+the "one" virtual storage was frozen.
+
+ >>> coordinatorB.close() # so we can reuse the transaction manager
+ >>> coordinator.sync()
+ >>> historical_coordinator = db.open(
+ ... transaction_manager=transaction2,
+ ... at=coordinator.root()['one']._p_serial)
+
+Way back then, the blob was merely happy, not ecstatic, and 'yrag' and 'nyrak'
+were not in the "nested: mapping.
+
+ >>> h_two = historical_coordinator.root()['two']
+ >>> h_conn = db.getVirtualConnection(h_two)
+ >>> h_conn.root()['nested']['blob'].open('r').read()
+ "I'm a happy Blob."
+ >>> 'nyrak' in h_conn.root()['nested']
+ False
+ >>> 'yrag' in h_conn.root()['nested']
+ False
+
+The storage is readonly, even though it is not frozen, because it is
+based on a historical, readonly coordinator connection.
+
+ >>> h_two._readonly
+ False
+ >>> h_two.isReadOnly()
+ True
+
+ >>> historical_coordinator.close()
+
+----
+TODO
+----
+
+Historical Base or Base from Another Database
+=============================================
+
+This should be possible. Stay tuned.
+
+.. ......... ..
+.. Footnotes ..
+.. ......... ..
+
+.. [#zc.vault] The ideas in zc.virtualstorage grow from zc.vault, which has
+ been used very successfully for similar use cases. However, zc.vault
+ requires significantly more developer knowledge and care when developing
+ applications that use it. In contrast, zc.virtualstorage should be very
+ natural and comfortable to ZODB developers, with very little extra to
+ learn.
+
+ On the other hand, zc.vault does currently have merge and resolution
+ stories that zc.virtualstorage does not yet have. zc.virtualstorage also
+ replaces low-level ZODB components such as the DB and Connection objects so
+ it is, at least arguably in some dimensions, riskier than zc.vault.
+
+ But the vastly improved developer story of zc.virtualstorage--which should
+ be so transparent that an already existing ZODB-based application could use
+ it without much modification--is intended to make zc.virtualstorage a
+ compelling replacement.
+
+.. [#get_is_recallable]
+
+ >>> conn0 is db.getVirtualConnection(zero)
+ True
+
+.. [#testcachecontrols] To examine these settings, we will look at internal
+ data structures.
+
+ Here's the cache size.
+
+ >>> db.getVirtualConnectionCacheSize()
+ 1000
+ >>> def check_cache_size():
+ ... size = db.getVirtualConnectionCacheSize()
+ ... for call in db.virtual_pool.all.as_weakref_list():
+ ... c = call()
+ ... if c is None:
+ ... continue
+ ... if c._cache.cache_size != size:
+ ... raise RuntimeError('hm, I have encountered an error')
+ ...
+ >>> check_cache_size()
+ >>> db.setVirtualConnectionCacheSize(1500)
+ >>> db.getVirtualConnectionCacheSize()
+ 1500
+ >>> check_cache_size()
+ >>> db.setVirtualConnectionCacheSize(500)
+ >>> db.getVirtualConnectionCacheSize()
+ 500
+ >>> check_cache_size()
+
+ Here's the timeout. Old connections are discarded when you open or close a
+ virtual connection.
+
+ >>> db.getVirtualConnectionTimeout()
+ 300
+ >>> db.setVirtualConnectionTimeout(200)
+ >>> db.getVirtualConnectionTimeout()
+ 200
+ >>> DB_module = __import__('ZODB.DB', globals(), locals(), ['chicken'])
+ >>> original_time = DB_module.time
+ >>> OFFSET = 0
+ >>> def stub_time():
+ ... return original_time() + OFFSET
+ ...
+ >>> DB_module.time = stub_time
+ >>> OFFSET = 200
+ >>> len(db.virtual_pool.all) # conn0, conn1, conn0B
+ 3
+ >>> bool(conn0._opened), bool(conn1._opened)
+ (True, True)
+ >>> bool(conn0B._opened)
+ False
+ >>> set(c() for c in db.virtual_pool.all.as_weakref_list()) == set((
+ ... conn0, conn1, conn0B))
+ True
+ >>> coordinator.close() # virtual closes and connects triggers cleanup
+ >>> len(db.virtual_pool.all) # conn0, conn1
+ 2
+ >>> set(c() for c in db.virtual_pool.all.as_weakref_list()) == set((
+ ... conn0, conn1))
+ True
+
+ Here's the count.
+
+ >>> print db.getVirtualConnectionPoolSize()
+ 3
+ >>> len(db.virtual_pool.all) # conn0, conn1
+ 2
+ >>> db.setVirtualConnectionPoolSize(1)
+ >>> db.getVirtualConnectionPoolSize()
+ 1
+ >>> len(db.virtual_pool.all)
+ 1
\ No newline at end of file
Property changes on: zc.virtualstorage/trunk/src/zc/virtualstorage/README.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: zc.virtualstorage/trunk/src/zc/virtualstorage/__init__.py
===================================================================
--- zc.virtualstorage/trunk/src/zc/virtualstorage/__init__.py (rev 0)
+++ zc.virtualstorage/trunk/src/zc/virtualstorage/__init__.py 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1 @@
+#
Added: zc.virtualstorage/trunk/src/zc/virtualstorage/base.py
===================================================================
--- zc.virtualstorage/trunk/src/zc/virtualstorage/base.py (rev 0)
+++ zc.virtualstorage/trunk/src/zc/virtualstorage/base.py 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,501 @@
+import logging
+import weakref
+from time import time # we do it this way so tests can monkeypatch stubs
+
+import persistent
+import persistent.mapping
+import ZODB.Connection
+import ZODB
+from ZODB.DB import KeyedConnectionPool
+import ZODB.interfaces
+import ZODB.POSException
+import ZODB.utils
+import BTrees
+import zope.interface
+
+BEGIN = 'BEGIN'
+COMMIT = 'COMMIT'
+VOTE = 'VOTE'
+FINISH = 'FINISH'
+
+logger = logging.getLogger('ZODB.DB.virtual')
+
+
+class Coordinator(ZODB.Connection.Connection):
+
+ _status = None
+
+ def close(self):
+ if not self._needs_to_join:
+ # We're currently joined to a transaction.
+ raise ConnectionStateError("Cannot close a connection joined to "
+ "a transaction")
+ self._db.closing(self) # let the virtual connections be closed first
+ ZODB.Connection.Connection.close(self)
+
+ def member_register(self, connection, obj):
+ if self._needs_to_join:
+ self.transaction_manager.get().join(self)
+ self._needs_to_join = False
+
+ def _coordinate_begin(self, transaction):
+ if self._status is None:
+ ZODB.Connection.Connection.tpc_begin(self, transaction)
+ self._status = BEGIN
+ self._virtual_tpc_members = set()
+ self._member_modified = []
+ else:
+ assert self._status == BEGIN
+
+ def member_tpc_begin(self, virtual_storage, transaction):
+ self._coordinate_begin(transaction)
+ self._virtual_tpc_members.add(virtual_storage)
+
+ def _coordinate_commit(self, transaction):
+ if not self._virtual_tpc_members and self._status == COMMIT:
+ # everyone has checked in. Now is the time to do a commit
+ # for the coordinator
+ ZODB.Connection.Connection.commit(self, transaction)
+
+ def member_committed(self, connection, transaction):
+ self._virtual_tpc_members.remove(connection._storage)
+ self._coordinate_commit(transaction)
+
+ def _coordinate_vote(self, transaction):
+ if self._status == COMMIT:
+ try:
+ vote = self._storage.tpc_vote
+ except AttributeError:
+ res = None
+ else:
+ res = vote(transaction)
+ self._vote_result = res
+ self._handle_serial(res)
+ self._status = VOTE
+ else:
+ assert self._status == VOTE
+ return self._vote_result
+
+ def member_tpc_vote(self, virtual_storage, transaction):
+ res = self._coordinate_vote(transaction)
+ self._virtual_tpc_members.add(virtual_storage)
+ return res
+
+ def _coordinate_finish(self, transaction):
+ if not self._virtual_tpc_members and self._status == FINISH:
+ # everyone has checked in. Now is the time to do a tpc_finish
+ # for the coordinator, sending in all functions
+ def callback(tid):
+ d = dict.fromkeys(self._modified)
+ for connection, modified in self._member_modified:
+ self._db.invalidate(tid, modified, connection)
+ self._db.invalidate(tid, d, self)
+ # It's important that the storage calls the passed function while it
+ # still has its lock. We don't want another thread to be able to read
+ # any updated data until we've had a chance to send an invalidation
+ # message to all of the other connections!
+ self._storage.tpc_finish(transaction, callback)
+ self._tpc_cleanup()
+
+ def member_tpc_finish(self, virtual_storage, transaction, connection,
+ modified):
+ self._member_modified.append((connection, modified))
+ self._virtual_tpc_members.remove(virtual_storage)
+ self._coordinate_finish(transaction)
+
+ def tpc_begin(self, transaction):
+ self._coordinate_begin(transaction)
+
+ def commit(self, transaction):
+ self._status = COMMIT
+ self._coordinate_commit(transaction)
+
+ def tpc_vote(self, transaction):
+ self._coordinate_vote(transaction)
+
+ def tpc_finish(self, transaction):
+ self._status = FINISH
+ self._coordinate_finish(transaction)
+
+ def _tpc_cleanup(self):
+ ZODB.Connection.Connection._tpc_cleanup(self)
+ self._status = None
+ self._member_modified = None
+ self._virtual_tpc_members = None
+ self._vote_result = None
+
+ # no changes to abort needed
+
+ def makeGhost(self, oid, p, serial):
+ obj = self._reader.getGhost(p)
+ self._pre_cache[oid] = obj
+ obj._p_oid = oid
+ obj._p_jar = self
+ obj._p_changed = None
+ obj._p_serial = serial
+ self._pre_cache.pop(oid)
+ self._cache[oid] = obj
+ return obj
+
+
+class VirtualDB(object):
+ def __init__(self, virtual_storage, db):
+ self._db = db
+ self._storage = virtual_storage
+ self.oid = virtual_storage._p_oid
+
+ @property
+ def databases(self):
+ return self._db.databases
+
+ @property
+ def database_name(self):
+ return '%s:zc.virtualstorage:%s' % (
+ self._db.database_name,
+ self._storage._p_oid)
+
+ @property
+ def classFactory(self):
+ return self._db.classFactory
+
+ def _returnToPool(self, connection):
+ assert connection._db is self
+ self._db.returnVirtualConnection(connection)
+
+
+class DB(ZODB.DB):
+ klass = Coordinator
+
+ def __init__(self, storage, pool_size=7, cache_size=400,
+ historical_pool_size=3, historical_cache_size=1000,
+ historical_timeout=300, database_name='unnamed',
+ databases=None, virtual_pool_size=3,
+ virtual_cache_size=1000, virtual_timeout=300):
+ ZODB.DB.__init__(self, storage, pool_size, cache_size,
+ historical_pool_size, historical_cache_size,
+ historical_timeout, database_name, databases)
+ self._virtual_cache_size = virtual_cache_size
+ self.virtual_pool = KeyedConnectionPool(
+ virtual_pool_size, virtual_timeout)
+ self._virtual_connection_map = weakref.WeakKeyDictionary()
+
+ def _connectionMap(self, f):
+ self._a()
+ try:
+ self.pool.map(f)
+ self.historical_pool.map(f)
+ self.virtual_pool.map(f)
+ finally:
+ self._r()
+
+ def getVirtualConnection(self, virtual_storage):
+ connection = virtual_storage._p_jar
+ if connection is None:
+ connection = ZODB.interfaces.IConnection(virtual_storage)
+ assert virtual_storage._p_oid is not None
+ assert isinstance(connection, Coordinator)
+ assert connection.db() is self
+ assert connection._opened
+ key = (virtual_storage._p_oid, connection.before)
+ pool = self.virtual_pool
+ self._a()
+ try:
+ map = self._virtual_connection_map.get(connection)
+ if map is None:
+ map = self._virtual_connection_map[connection] = (
+ weakref.WeakValueDictionary())
+ res = map.get(virtual_storage._p_oid)
+ if res is None:
+ res = pool.pop(key)
+ if res is None:
+ res = VirtualConnection(
+ VirtualDB(virtual_storage, self),
+ self.getVirtualConnectionCacheSize(),
+ connection.before)
+ pool.push(res, key)
+ res = pool.pop(key)
+ assert res is not None
+ else:
+ res._storage = res._db._storage = virtual_storage
+ map[virtual_storage._p_oid] = res
+ if not res._opened:
+ res.open(connection.transaction_manager)
+ pool.availableGC()
+ return res
+ finally:
+ self._r()
+
+ def closing(self, coordinator):
+ self._a()
+ try:
+ map = self._virtual_connection_map.get(coordinator)
+ if map is not None:
+ for connection in map.values():
+ connection.close()
+ assert connection._storage is connection._db._storage is None
+ del self._virtual_connection_map[coordinator]
+ finally:
+ self._r()
+
+ def returnVirtualConnection(self, connection):
+ assert isinstance(connection, VirtualConnection)
+ storage = connection._storage
+ key = (storage._p_oid, connection.before)
+ self._a()
+ try:
+ coordinator = storage._p_jar
+ assert coordinator._db is self
+ connection._opened = None
+ am = self._activity_monitor
+ if am is not None:
+ am.closedConnection(connection)
+ map = self._virtual_connection_map[coordinator]
+ del map[storage._p_oid]
+ connection._storage = connection._db._storage = None
+ self.virtual_pool.repush(connection, key)
+ finally:
+ self._r()
+
+ def getVirtualConnectionCacheSize(self):
+ return self._virtual_cache_size
+
+ def setVirtualConnectionCacheSize(self, size):
+ self._virtual_cache_size = size
+ def setsize(c):
+ c._cache.cache_size = size
+ self._a()
+ try:
+ self.virtual_pool.map(setsize)
+ finally:
+ self._r()
+
+ def getVirtualConnectionTimeout(self):
+ return self.virtual_pool.timeout
+
+ def setVirtualConnectionTimeout(self, timeout):
+ self._a()
+ try:
+ self.virtual_pool.timeout = timeout
+ finally:
+ self._r()
+
+ def getVirtualConnectionPoolSize(self):
+ return self.virtual_pool.size
+
+ def setVirtualConnectionPoolSize(self, size):
+ self._a()
+ try:
+ self.virtual_pool.size = size
+ finally:
+ self._r()
+
+
+class VirtualConnection(ZODB.Connection.Connection):
+
+ def __init__(self, db, cache_size=400, before=None):
+ ZODB.Connection.Connection.__init__(self, db, cache_size, before)
+ self._raw_invalidated = set()
+
+ def _register(self, obj=None):
+ self._normal_storage._p_jar.member_register(self, obj)
+ ZODB.Connection.Connection._register(self, obj)
+
+ def commit(self, transaction):
+ ZODB.Connection.Connection.commit(self, transaction)
+ self._normal_storage._p_jar.member_committed(self, transaction)
+
+ def tpc_finish(self, transaction):
+ self._normal_storage.tpc_finish(transaction, self)
+ self._tpc_cleanup()
+
+ def invalidate(self, tid, oids):
+ if self.before is not None:
+ # this is an historical connection. Invalidations are irrelevant.
+ return
+ getInvalidatedOID = self._normal_storage.getInvalidatedOID
+ self._inv_lock.acquire()
+ try:
+ if self._txn_time is None:
+ self._txn_time = tid
+ if self._opened is not None:
+ mapped = [oid for oid in
+ (getInvalidatedOID(o) for o in oids)
+ if oid is not None]
+ self._invalidated.update(mapped)
+ else:
+ # invalidations while closed are not mapped (because accessing
+ # the virtual storage while its connection is closed will
+ # generate an error)
+ self._raw_invalidated.update(oids)
+ finally:
+ self._inv_lock.release()
+
+ # Process pending invalidations.
+ def _flush_invalidations(self):
+ getInvalidatedOID = self._normal_storage.getInvalidatedOID
+ self._inv_lock.acquire()
+ try:
+ # see comments in ZODB.Connection.Connection._flush_invalidations.
+ # we override this to use the _raw_invalidated values from the
+ # ``invalidate`` method above.
+ if self._invalidatedCache:
+ self._invalidatedCache = False
+ invalidated = self._cache.cache_data.copy()
+ else:
+ if self._raw_invalidated:
+ self._invalidated.update(
+ oid for oid in
+ (getInvalidatedOID(o) for o in self._raw_invalidated)
+ if oid is not None)
+ invalidated = dict.fromkeys(self._invalidated)
+ self._raw_invalidated.clear()
+ self._invalidated = set()
+ self._txn_time = None
+ finally:
+ self._inv_lock.release()
+ self._cache.invalidate(invalidated)
+ # Now is a good time to collect some garbage.
+ self._cache.incrgc()
+
+
+class VirtualStorage(persistent.Persistent):
+
+ zope.interface.implements(ZODB.interfaces.IBlobStorage)
+
+ def __init__(self, base=None):
+ if base is not None and not base.isReadOnly():
+ raise ValueError('base must be read only')
+ self.base = base
+ self.local = BTrees.OOBTree.BTree()
+ self.reverse_local = BTrees.OOBTree.BTree() # needed for invalidations
+ self.new = BTrees.OOBTree.TreeSet()
+ self.bucket = BTrees.OOBTree.BTree() # real oid to effective oid, obj
+ self._readonly = False
+ self._giveRootOID = True
+
+ def temporaryDirectory(self):
+ return self._p_jar._storage.temporaryDirectory()
+
+ def getSize(self):
+ return self._p_jar._storage.getSize() # shrug.
+
+ def freeze(self):
+ if self._readonly:
+ raise ValueError('already frozen')
+ self._readonly = True
+
+ def sortKey(self):
+ return '.'.join((self._p_jar.sortKey(), self._p_oid))
+ getName = sortKey
+
+ def getInvalidatedOID(self, oid, default=None):
+ if oid in self.new:
+ return oid
+ return self.reverse_local.get(oid, default)
+
+ def isReadOnly(self):
+ return self._readonly or self._p_jar.isReadOnly()
+
+ def load(self, oid, version=''):
+ if version != '':
+ raise ValueError('no versions allowed')
+ real = self.local.get(oid)
+ if real is None:
+ if oid in self.new:
+ real = oid
+ elif self.base is not None:
+ return self.base.load(oid)
+ else:
+ raise ZODB.POSException.POSKeyError(oid)
+ return self._p_jar._storage.load(real, version)
+
+ def loadBefore(self, oid, serial):
+ # note that our views on our data structures are already MVCC.
+ real = self.local.get(oid)
+ if real is None:
+ if oid in self.new:
+ real = oid
+ elif self.base is not None:
+ return self.base.loadBefore(oid, serial)
+ else:
+ raise ZODB.POSException.POSKeyError(oid)
+ return self._p_jar._storage.loadBefore(real, serial)
+
+ def loadBlob(self, oid, serial):
+ real = self.local.get(oid)
+ if real is None:
+ if oid in self.new:
+ real = oid
+ elif self.base is not None:
+ return self.base.loadBlob(oid, serial)
+ else:
+ raise ZODB.POSException.POSKeyError(oid)
+ return self._p_jar._storage.loadBlob(real, serial)
+
+ def new_oid(self):
+ if self._giveRootOID and self.base is None and not self.new:
+ res = ZODB.utils.z64
+ self._giveRootOID = False
+ else:
+ res = self._p_jar.new_oid()
+ self.new.insert(res)
+ return res
+
+ def store(self, oid, oldserial, data, version, transaction):
+ assert version == ''
+ return self._store(
+ oid, data,
+ lambda real: self._p_jar._storage.store(
+ real, oldserial, data, '', transaction))
+
+ def storeBlob(self, oid, oldserial, data, blobfilename, version,
+ transaction):
+ assert version == ''
+ return self._store(
+ oid, data,
+ lambda real: self._p_jar._storage.storeBlob(
+ real, oldserial, data, blobfilename, '', transaction))
+
+ def _store(self, oid, data, callable):
+ if oid in self.local:
+ real = self.local[oid]
+ elif oid in self.new:
+ real = oid
+ else:
+ real = self._p_jar.new_oid()
+ self.local[oid] = real
+ self.reverse_local[real] = oid
+ serial = callable(real)
+ if oid not in self.bucket:
+ # prevent packing away
+ self.bucket[oid] = self._p_jar.makeGhost(real, data, serial)
+ return serial
+
+ def tpc_begin(self, transaction):
+ if self.isReadOnly():
+ raise ZODB.POSException.ReadOnlyError()
+ self._p_jar.member_tpc_begin(self, transaction)
+
+ def tpc_vote(self, transaction):
+ res = self._p_jar.member_tpc_vote(self, transaction)
+ if res:
+ res = [
+ (local, serial) for local, serial in (
+ (self.getInvalidatedOID(oid), serial) for oid, serial in
+ res)
+ if local is not None]
+ return res
+
+ def _getRealOID(self, oid):
+ if oid in self.new:
+ return oid
+ return self.local[oid] # or else it is an error, even if it is in base
+
+ def tpc_finish(self, transaction, connection):
+ modified = dict.fromkeys(
+ self._getRealOID(oid) for oid in connection._modified)
+ self._p_jar.member_tpc_finish(self, transaction, connection, modified)
+
+ def tpc_abort(self, transaction):
+ pass # normal invalidations in coordinator of modified objects are
+ # sufficient to make `local` `reverse_local` and `bucket` fine.
Added: zc.virtualstorage/trunk/src/zc/virtualstorage/component.xml
===================================================================
--- zc.virtualstorage/trunk/src/zc/virtualstorage/component.xml (rev 0)
+++ zc.virtualstorage/trunk/src/zc/virtualstorage/component.xml 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,23 @@
+<component prefix="zc.virtualstorage.config">
+
+ <sectiontype name="zodb_with_virtualstorage_support" datatype=".Database"
+ implements="ZODB.database" extends="zodb">
+
+ <key name="virtual-cache-size" datatype="integer" default="1000"/>
+ <description>
+ Target size, in number of objects, of each virtual connection's
+ object cache.
+ </description>
+ <key name="virtual-timeout" datatype="time-interval" default="5m"/>
+ <description>
+ The minimum interval that an unused virtual connection should be
+ kept.
+ </description>
+ <key name="virtual-pool-size" datatype="integer" default="3"/>
+ <description>
+ The expected maximum total number of virtual connections
+ simultaneously open.
+ </description>
+ </sectiontype>
+
+</component>
\ No newline at end of file
Added: zc.virtualstorage/trunk/src/zc/virtualstorage/config.py
===================================================================
--- zc.virtualstorage/trunk/src/zc/virtualstorage/config.py (rev 0)
+++ zc.virtualstorage/trunk/src/zc/virtualstorage/config.py 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,24 @@
+import ZODB.config
+import zc.virtualstorage.base
+
+class Database(ZODB.config.BaseConfig):
+
+ def open(self, databases=None):
+ section = self.config
+ storage = section.storage.open()
+ try:
+ return zc.virtualstorage.base.DB(storage,
+ pool_size=section.pool_size,
+ cache_size=section.cache_size,
+ historical_pool_size=section.historical_pool_size,
+ historical_cache_size=section.historical_cache_size,
+ historical_timeout=section.historical_timeout,
+ database_name=section.database_name,
+ databases=databases,
+ virtual_pool_size=section.virtual_pool_size,
+ virtual_cache_size=section.virtual_cache_size,
+ virtual_timeout=section.virtual_timeout,
+ )
+ except:
+ storage.close()
+ raise
\ No newline at end of file
Added: zc.virtualstorage/trunk/src/zc/virtualstorage/tests.py
===================================================================
--- zc.virtualstorage/trunk/src/zc/virtualstorage/tests.py (rev 0)
+++ zc.virtualstorage/trunk/src/zc/virtualstorage/tests.py 2007-12-02 11:22:38 UTC (rev 82076)
@@ -0,0 +1,44 @@
+##############################################################################
+#
+# Copyright (c) 2007 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""
+$Id$
+"""
+import unittest
+from zope.testing import doctest, module
+
+def setUp(test):
+ module.setUp(test, 'virtualstorage_txt')
+
+def tearDown(test):
+ test.globs['db'].close()
+ #test.globs['db2'].close()
+ test.globs['blob_storage'].close()
+ test.globs['storage'].cleanup()
+ # the DB class masks the module because of __init__ shenanigans
+ DB_module = __import__('ZODB.DB', globals(), locals(), ['chicken'])
+ DB_module.time = test.globs['original_time']
+ module.tearDown(test)
+
+def test_suite():
+ return unittest.TestSuite((
+ doctest.DocFileSuite('README.txt',
+ setUp=setUp,
+ tearDown=tearDown,
+ optionflags=doctest.INTERPRET_FOOTNOTES,
+ ),
+ ))
+
+if __name__ == '__main__':
+ unittest.main(defaultTest='test_suite')
+
More information about the Checkins
mailing list