[Checkins] SVN: z3c.vcsync/trunk/src/z3c/vcsync/ Some API changes
for import/export functionality. A much nicer doctest
Martijn Faassen
faassen at infrae.com
Mon Apr 21 14:53:26 EDT 2008
Log message for revision 85566:
Some API changes for import/export functionality. A much nicer doctest
for it too.
Changed:
U z3c.vcsync/trunk/src/z3c/vcsync/README.txt
U z3c.vcsync/trunk/src/z3c/vcsync/__init__.py
U z3c.vcsync/trunk/src/z3c/vcsync/importexport.py
A z3c.vcsync/trunk/src/z3c/vcsync/importexport.txt
U z3c.vcsync/trunk/src/z3c/vcsync/svn.txt
U z3c.vcsync/trunk/src/z3c/vcsync/tests.py
-=-
Modified: z3c.vcsync/trunk/src/z3c/vcsync/README.txt
===================================================================
--- z3c.vcsync/trunk/src/z3c/vcsync/README.txt 2008-04-21 18:37:02 UTC (rev 85565)
+++ z3c.vcsync/trunk/src/z3c/vcsync/README.txt 2008-04-21 18:53:25 UTC (rev 85566)
@@ -818,210 +818,3 @@
>>> container2['alpha'].payload
4000
-Exporting
----------
-
-Besides version control synchronization, the same infrastructure
-can also be used for an export and import procedure.
-
-Here we export the state to a target directory::
-
- >>> from z3c.vcsync import export_state
- >>> target = create_test_dir()
- >>> export_state(TestState(container2), target)
-
-Let's inspect the state of container2 first::
-
- >>> sorted(container2.keys())
- ['alpha', 'foo', 'hoi', 'sub']
- >>> sorted(container2['sub'].keys())
- ['qux']
- >>> container2['alpha'].payload
- 4000
- >>> container2['foo'].payload
- 1
- >>> container2['hoi'].payload
- 3000
- >>> container2['sub']['qux'].payload
- 3
-
-We can inspect the target directory and see the data is there::
-
- >>> sorted([p.basename for p in target.listdir()])
- ['alpha.test', 'foo.test', 'hoi.test', 'sub']
- >>> target.join('alpha.test').read()
- '4000\n'
- >>> target.join('foo.test').read()
- '1\n'
- >>> target.join('hoi.test').read()
- '3000\n'
- >>> sub = target.join('sub')
- >>> sorted([p.basename for p in sub.listdir()])
- ['qux.test']
- >>> sub.join('qux.test').read()
- '3\n'
-
-We can also export to a zipfile. Let's first add an empty folder to the
-content to make things more difficult::
-
- >>> container2['empty'] = Container()
-
-Now let's do the export::
-
- >>> ziptarget = create_test_dir()
- >>> zipfile_path = ziptarget.join('export.zip')
- >>> from z3c.vcsync import export_state_zip
- >>> export_state_zip(TestState(container2), 'data', zipfile_path)
-
-Inspecting the zipfile shows us the right files::
-
- >>> from zipfile import ZipFile
- >>> zf = ZipFile(zipfile_path.strpath, 'r')
- >>> sorted(zf.namelist())
- ['data/', 'data/alpha.test', 'data/empty/', 'data/foo.test',
- 'data/hoi.test', 'data/sub/', 'data/sub/qux.test']
- >>> zf.read('data/alpha.test')
- '4000\n'
- >>> zf.read('data/foo.test')
- '1\n'
- >>> zf.read('data/hoi.test')
- '3000\n'
- >>> zf.read('data/sub/qux.test')
- '3\n'
-
-Importing
----------
-
-Now we import the state on the filesystem into a fresh container.
-
- >>> container3 = Container()
- >>> container3.__name__ = 'root'
-
- >>> from z3c.vcsync import import_state
-
-We pass in a special function that prints out all the objects we just
-imported::
-
- >>> def f(obj):
- ... print obj
- >>> import_state(TestState(container3), target, modified_function=f)
- <Item object at ...>
- <Item object at ...>
- <z3c.vcsync.tests.Container object at ...>
- <Item object at ...>
- <Item object at ...>
-
-
-We expect the structure to be the same as what we exported::
-
- >>> sorted(container3.keys())
- ['alpha', 'foo', 'hoi', 'sub']
- >>> sorted(container3['sub'].keys())
- ['qux']
- >>> container3['alpha'].payload
- 4000
- >>> container3['foo'].payload
- 1
- >>> container3['hoi'].payload
- 3000
- >>> container3['sub']['qux'].payload
- 3
-
-We can also import from a zipfile::
-
- >>> container4 = Container()
- >>> container4.__name__ = 'root'
-
- >>> from z3c.vcsync import import_state_zip
- >>> import_state_zip(TestState(container4), 'data', zipfile_path)
-
-We expect the structure to be the same as what we exported (including the
-empty folder)::
-
- >>> sorted(container4.keys())
- ['alpha', 'empty', 'foo', 'hoi', 'sub']
- >>> sorted(container4['sub'].keys())
- ['qux']
- >>> container4['alpha'].payload
- 4000
- >>> container4['foo'].payload
- 1
- >>> container4['hoi'].payload
- 3000
- >>> container4['sub']['qux'].payload
- 3
- >>> sorted(container4['empty'].keys())
- []
-
-Importing into existing content
--------------------------------
-
-We can also import into a container that already has existing
-content. In this case any existing state is left alone (never
-overwritten). New content is added however. Let's add a small export
-to demonstrate this. We will later try to load it into container4::
-
- >>> container5 = Container()
- >>> container5.__name__ = 'root'
-
-Our new export contains new item, which should be added::
-
- >>> container5['new_item'] = Item(7777)
-
-It will also contain an item ``alpha``. Loading this should not
-overwrite the item ``alpha`` we already have in container::
-
- >>> container5['alpha'] = Item(5000)
-
-We will also add a new sub container, which should up::
-
- >>> container5['subextra'] = Container()
- >>> container5['subextra']['new_too'] = Item(8888)
-
-We will also add a new item to an existing sub container (``sub``)::
-
- >>> container5['sub'] = Container()
- >>> container5['sub']['new_as_well'] = Item(9999)
-
-Finally we we will try to add an object to something that's not a container
-in the original structure. This attempt should also be ignored::
-
- >>> container5['foo'] = Container()
- >>> container5['foo']['heh'] = Item(4444)
-
-Now let's turn this into a zip export::
-
- >>> ziptarget = create_test_dir()
- >>> zipfile_path = ziptarget.join('export.zip')
- >>> export_state_zip(TestState(container5), 'data', zipfile_path)
-
-We will now import this new zipfile into container4::
-
- >>> import_state_zip(TestState(container4), 'data', zipfile_path)
-
-We expect the original content to be still there, even in case of ``alpha``::
-
- >>> container4['alpha'].payload
- 4000
- >>> container4['foo'].payload
- 1
- >>> container4['hoi'].payload
- 3000
- >>> container4['sub']['qux'].payload
- 3
- >>> sorted(container4['empty'].keys())
- []
-
-We expect to see the new content in the containers::
-
- >>> sorted(container4.keys())
- ['alpha', 'empty', 'foo', 'hoi', 'new_item', 'sub', 'subextra']
- >>> container4['new_item'].payload
- 7777
- >>> sorted(container4['sub'].keys())
- ['new_as_well', 'qux']
- >>> container4['sub']['new_as_well'].payload
- 9999
- >>> sorted(container4['subextra'].keys())
- ['new_too']
-
Modified: z3c.vcsync/trunk/src/z3c/vcsync/__init__.py
===================================================================
--- z3c.vcsync/trunk/src/z3c/vcsync/__init__.py 2008-04-21 18:37:02 UTC (rev 85565)
+++ z3c.vcsync/trunk/src/z3c/vcsync/__init__.py 2008-04-21 18:53:25 UTC (rev 85566)
@@ -1,3 +1,3 @@
from vc import Synchronizer
-from importexport import (export_state, import_state,
- export_state_zip, import_state_zip)
+from importexport import (export, import_,
+ export_zip, import_zip)
Modified: z3c.vcsync/trunk/src/z3c/vcsync/importexport.py
===================================================================
--- z3c.vcsync/trunk/src/z3c/vcsync/importexport.py 2008-04-21 18:37:02 UTC (rev 85565)
+++ z3c.vcsync/trunk/src/z3c/vcsync/importexport.py 2008-04-21 18:53:25 UTC (rev 85566)
@@ -4,58 +4,55 @@
from zope.app.container.interfaces import IContainer
from z3c.vcsync.interfaces import IVcDump, IVcFactory
from zope.component import getUtility
-
-def export_state(state, path):
- export_helper(state.root, path)
-def export_helper(obj, path):
- for obj in obj.values():
+def export(root, path):
+ for obj in root.values():
IVcDump(obj).save(path)
if IContainer.providedBy(obj):
- export_helper(obj, path.join(obj.__name__))
+ export(obj, path.join(obj.__name__))
-def export_state_zip(state, name, zippath):
+def export_zip(root, name, zippath):
tmp_dir = py.path.local(tempfile.mkdtemp())
try:
save_path = tmp_dir.join(name)
save_path.ensure(dir=True)
- export_state(state, save_path)
+ export(root, save_path)
zf = zipfile.ZipFile(zippath.strpath, 'w')
- export_state_zip_helper(zf, tmp_dir, save_path)
+ _export_zip_helper(zf, tmp_dir, save_path)
zf.close()
finally:
tmp_dir.remove()
-def export_state_zip_helper(zf, save_path, path):
+def _export_zip_helper(zf, save_path, path):
if path.check(dir=True):
zf.writestr(path.relto(save_path) + '/', '')
for p in path.listdir():
- export_state_zip_helper(zf, save_path, p)
+ _export_zip_helper(zf, save_path, p)
else:
zf.write(path.strpath, path.relto(save_path))
-def import_state(state, path, modified_function=None):
- modified_objects = import_helper(state.root, path)
+def import_(root, path, modified_function=None):
+ modified_objects = _import_helper(root, path)
if modified_function is not None:
for obj in modified_objects:
modified_function(obj)
-def import_helper(obj, path):
+def _import_helper(obj, path):
modified_objects = []
for p in path.listdir():
factory = getUtility(IVcFactory, name=p.ext)
name = p.purebasename
if name not in obj:
obj[name] = new_obj = factory(p)
+ modified_objects.append(new_obj)
else:
new_obj = obj[name]
- modified_objects.append(new_obj)
if p.check(dir=True) and IContainer.providedBy(new_obj):
- r = import_helper(new_obj, p)
+ r = _import_helper(new_obj, p)
modified_objects.extend(r)
return modified_objects
-def import_state_zip(state, name, zippath, modified_function=None):
+def import_zip(root, name, zippath, modified_function=None):
tmp_dir = py.path.local(tempfile.mkdtemp())
try:
zf = zipfile.ZipFile(zippath.strpath, 'r')
@@ -71,8 +68,8 @@
new_path.write(zf.read(p))
else:
new_path.ensure(dir=True)
- import_state(state, tmp_dir.join(name),
- modified_function=modified_function)
+ import_(root, tmp_dir.join(name),
+ modified_function=modified_function)
zf.close()
finally:
tmp_dir.remove()
Added: z3c.vcsync/trunk/src/z3c/vcsync/importexport.txt
===================================================================
--- z3c.vcsync/trunk/src/z3c/vcsync/importexport.txt (rev 0)
+++ z3c.vcsync/trunk/src/z3c/vcsync/importexport.txt 2008-04-21 18:53:25 UTC (rev 85566)
@@ -0,0 +1,294 @@
+Importing and exporting content
+===============================
+
+This package can do more than synchronize content with SVN: its
+infrastructure is also for exporting object structures to the
+filesystem or a zip file, and importing them back again.
+
+A simple item to export
+-----------------------
+
+We define a very simple content object we want to export::
+
+ >>> class Item(object):
+ ... def __init__(self, payload):
+ ... self.payload = payload
+ ... def __repr__(self):
+ ... return "<Item %s with payload %s>" % (self.__name__, self.payload)
+
+We have already defined a ``Container`` object before.
+
+Let's set up some objects to export::
+
+ >>> data = Container()
+ >>> data.__name__ = 'root'
+ >>> data['alpha'] = Item(4000)
+ >>> data['foo'] = Item(1)
+ >>> data['hoi'] = Item(3000)
+ >>> data['sub'] = Container()
+ >>> data['sub']['qux'] = Item(3)
+ >>> data['empty'] = Container()
+
+Writing to the filesystem
+-------------------------
+
+First we need to grok this package itself (normally done when you use
+it)::
+
+ >>> import grok.testing
+ >>> grok.testing.grok('z3c.vcsync')
+
+In order to be able to export instances of ``Item``, we need to set up
+an ``ISerializer``::
+
+ >>> import grok
+ >>> from z3c.vcsync.interfaces import ISerializer
+ >>> class ItemSerializer(grok.Adapter):
+ ... grok.provides(ISerializer)
+ ... grok.context(Item)
+ ... def serialize(self, f):
+ ... f.write(str(self.context.payload))
+ ... f.write('\n')
+ ... def name(self):
+ ... return self.context.__name__ + '.test'
+ >>> grok.testing.grok_component('ItemSerializer', ItemSerializer)
+ True
+
+Exporting
+---------
+
+Now we can export our tree to the filesystem by passing the root
+object to the ``export`` function:
+
+ >>> from z3c.vcsync import export
+ >>> target = create_test_dir()
+ >>> export(data, target)
+
+The data is now exported to the target directory::
+
+ >>> sorted([p.basename for p in target.listdir()])
+ ['alpha.test', 'empty', 'foo.test', 'hoi.test', 'sub']
+ >>> target.join('alpha.test').read()
+ '4000\n'
+ >>> target.join('foo.test').read()
+ '1\n'
+ >>> target.join('hoi.test').read()
+ '3000\n'
+ >>> sub = target.join('sub')
+ >>> sorted([p.basename for p in sub.listdir()])
+ ['qux.test']
+ >>> sub.join('qux.test').read()
+ '3\n'
+ >>> empty = target.join('empty')
+ >>> sorted([p.basename for p in empty.listdir()])
+ []
+
+Exporting to a zipfile
+----------------------
+
+We can also export our content to a zipfile::
+
+ >>> from z3c.vcsync import export_zip
+ >>> ziptarget = create_test_dir()
+ >>> zipfile_path = ziptarget.join('export.zip')
+ >>> export_zip(data, 'data', zipfile_path)
+
+Inspecting the zipfile shows us the right files::
+
+ >>> from zipfile import ZipFile
+ >>> zf = ZipFile(zipfile_path.strpath, 'r')
+ >>> sorted(zf.namelist())
+ ['data/', 'data/alpha.test', 'data/empty/', 'data/foo.test',
+ 'data/hoi.test', 'data/sub/', 'data/sub/qux.test']
+ >>> zf.read('data/alpha.test')
+ '4000\n'
+ >>> zf.read('data/foo.test')
+ '1\n'
+ >>> zf.read('data/hoi.test')
+ '3000\n'
+ >>> zf.read('data/sub/qux.test')
+ '3\n'
+
+Reading from the filesystem
+---------------------------
+
+Before we can import, we need to register two utilities, one to create
+new ``Item`` instances and the other to create new ``Container``
+instances. Note that an ``IParser`` implemenentation is not
+necessary, though of course it can't hurt to reuse it the factory if
+you happen to have it anyway (as we it is needed for synchronization).
+
+The factory to create the item::
+
+ >>> from z3c.vcsync.interfaces import IVcFactory
+ >>> class ItemFactory(grok.GlobalUtility):
+ ... grok.provides(IVcFactory)
+ ... grok.name('.test')
+ ... def __call__(self, path):
+ ... payload = int(path.read())
+ ... return Item(payload)
+ >>> grok.testing.grok_component('ItemFactory', ItemFactory)
+ True
+
+The factory to create the container::
+
+ >>> class ContainerFactory(grok.GlobalUtility):
+ ... grok.provides(IVcFactory)
+ ... def __call__(self, path):
+ ... return Container()
+ >>> grok.testing.grok_component('ContainerFactory', ContainerFactory)
+ True
+
+Importing
+---------
+
+We prepare a new object to import into::
+
+ >>> new_root = Container()
+ >>> new_root.__name__ = 'root'
+
+We can optionally pass in a function that does something with all
+objects we just imported. Let's do that here, just printing out the
+object::
+
+ >>> def f(obj):
+ ... print obj
+
+We will import from the filesystem structure that we previously
+exported to. When we do the import, we see each object we are
+importing being printed::
+
+ >>> from z3c.vcsync import import_
+ >>> import_(new_root, target, modified_function=f)
+ <Item alpha with payload 4000>
+ <Item foo with payload 1>
+ <Container sub>
+ <Item qux with payload 3>
+ <Container empty>
+ <Item hoi with payload 3000>
+
+The structure under new_root is now the same as what we exported
+before:::
+
+ >>> sorted(new_root.keys())
+ ['alpha', 'empty', 'foo', 'hoi', 'sub']
+ >>> sorted(new_root['sub'].keys())
+ ['qux']
+ >>> new_root['alpha'].payload
+ 4000
+ >>> new_root['foo'].payload
+ 1
+ >>> new_root['hoi'].payload
+ 3000
+ >>> new_root['sub']['qux'].payload
+ 3
+
+Importing from a zipfile
+------------------------
+
+We can also import from the zipfile we created before::
+
+ >>> zip_root = Container()
+ >>> zip_root.__name__ = 'root'
+
+ >>> from z3c.vcsync import import_zip
+ >>> import_zip(zip_root, 'data', zipfile_path)
+
+We expect the structure to be the same as what we exported::
+
+ >>> sorted(zip_root.keys())
+ ['alpha', 'empty', 'foo', 'hoi', 'sub']
+ >>> sorted(zip_root['sub'].keys())
+ ['qux']
+ >>> zip_root['alpha'].payload
+ 4000
+ >>> zip_root['foo'].payload
+ 1
+ >>> zip_root['hoi'].payload
+ 3000
+ >>> zip_root['sub']['qux'].payload
+ 3
+ >>> zip_root['empty'].keys()
+ []
+
+Importing into existing content
+-------------------------------
+
+We can also import into a container that already has existing
+content. Existing objects are never overwritten, but new containers and objects
+are added to this tree.
+
+Let's add create a new export to demonstrate this. We will later try
+to load it into ``zip_root``::
+
+ >>> extra = Container()
+ >>> extra.__name__ = 'root'
+
+Our new export contains a new item, which should be added::
+
+ >>> extra['new_item'] = Item(7777)
+
+It will contains an item ``alpha``. Loading this should not overwrite
+the item ``alpha`` we already have in container::
+
+ >>> extra['alpha'] = Item(5000)
+
+We will also add a new sub container, which should appear::
+
+ >>> extra['subextra'] = Container()
+ >>> extra['subextra']['new_too'] = Item(8888)
+
+We will also add a new item to an existing sub container (``sub``)::
+
+ >>> extra['sub'] = Container()
+ >>> extra['sub']['new_as_well'] = Item(9999)
+
+Finally we we will try to add an object to something that's not a
+actually a container (namely ``foo``, an item) in the original
+structure. This attempt should also be ignored::
+
+ >>> extra['foo'] = Container()
+ >>> extra['foo']['heh'] = Item(4444)
+
+Let's turn extra into a zip export::
+
+ >>> ziptarget = create_test_dir()
+ >>> zipfile_path = ziptarget.join('export.zip')
+ >>> export_zip(extra, 'data', zipfile_path)
+
+We will now import this new zip file into ``zip_root``, using the
+modified_function to print objects::
+
+ >>> import_zip(zip_root, 'data', zipfile_path, modified_function=f)
+ <Container subextra>
+ <Item new_too with payload 8888>
+ <Item new_item with payload 7777>
+ <Item new_as_well with payload 9999>
+
+Note that only those objects *new* to the tree will be printed, not all the
+objects in the original zip file.
+
+The original content is still there, not overwritten, even in case of
+``alpha``::
+
+ >>> zip_root['alpha'].payload
+ 4000
+ >>> zip_root['foo'].payload
+ 1
+ >>> zip_root['hoi'].payload
+ 3000
+ >>> zip_root['sub']['qux'].payload
+ 3
+
+The completey new content has also appeared::
+
+ >>> sorted(zip_root.keys())
+ ['alpha', 'empty', 'foo', 'hoi', 'new_item', 'sub', 'subextra']
+ >>> zip_root['new_item'].payload
+ 7777
+ >>> sorted(zip_root['sub'].keys())
+ ['new_as_well', 'qux']
+ >>> zip_root['sub']['new_as_well'].payload
+ 9999
+ >>> sorted(zip_root['subextra'].keys())
+ ['new_too']
Modified: z3c.vcsync/trunk/src/z3c/vcsync/svn.txt
===================================================================
--- z3c.vcsync/trunk/src/z3c/vcsync/svn.txt 2008-04-21 18:37:02 UTC (rev 85565)
+++ z3c.vcsync/trunk/src/z3c/vcsync/svn.txt 2008-04-21 18:53:25 UTC (rev 85566)
@@ -99,7 +99,8 @@
>>> grok.testing.grok('z3c.vcsync')
We need to provide a serializer for the Item class that takes an item
-and writes it to the filesystem::
+and writes it to the filesystem to a file with a particular extension
+(``.test``)::
>>> import grok
>>> from z3c.vcsync.interfaces import ISerializer
Modified: z3c.vcsync/trunk/src/z3c/vcsync/tests.py
===================================================================
--- z3c.vcsync/trunk/src/z3c/vcsync/tests.py 2008-04-21 18:37:02 UTC (rev 85565)
+++ z3c.vcsync/trunk/src/z3c/vcsync/tests.py 2008-04-21 18:53:25 UTC (rev 85566)
@@ -86,6 +86,9 @@
def __delitem__(self, name):
del self._data[name]
+ def __repr__(self):
+ return "<Container %s>" % (self.__name__)
+
def svn_repo_wc():
"""Create an empty SVN repository.
@@ -146,6 +149,13 @@
optionflags=doctest.ELLIPSIS + doctest.NORMALIZE_WHITESPACE),
doctest.DocFileSuite(
+ 'importexport.txt',
+ setUp=setUpZope,
+ tearDown=cleanUpZope,
+ globs=globs,
+ optionflags=doctest.ELLIPSIS + doctest.NORMALIZE_WHITESPACE),
+
+ doctest.DocFileSuite(
'svn.txt',
setUp=setUpZope,
tearDown=cleanUpZope,
More information about the Checkins
mailing list