[Checkins] SVN: zope.app.fssync/trunk/ Merge jim-hack branch to the trunk (-r 93648:97911).
Amos Latteier
amos at latteier.com
Wed Mar 11 16:04:23 EDT 2009
Log message for revision 97912:
Merge jim-hack branch to the trunk (-r 93648:97911).
Changed:
_U zope.app.fssync/trunk/
A zope.app.fssync/trunk/CHANGES.txt
U zope.app.fssync/trunk/README.txt
U zope.app.fssync/trunk/buildout.cfg
U zope.app.fssync/trunk/setup.py
U zope.app.fssync/trunk/src/zope/app/fssync/browser/__init__.py
A zope.app.fssync/trunk/src/zope/app/fssync/demo_server.py
U zope.app.fssync/trunk/src/zope/app/fssync/fssync.py
U zope.app.fssync/trunk/src/zope/app/fssync/fssync.txt
U zope.app.fssync/trunk/src/zope/app/fssync/ftesting.zcml
U zope.app.fssync/trunk/src/zope/app/fssync/ftests.py
U zope.app.fssync/trunk/src/zope/app/fssync/main.py
A zope.app.fssync/trunk/src/zope/app/fssync/merge.py
A zope.app.fssync/trunk/src/zope/app/fssync/merge.txt
U zope.app.fssync/trunk/src/zope/app/fssync/passwd.py
U zope.app.fssync/trunk/src/zope/app/fssync/security.txt
A zope.app.fssync/trunk/src/zope/app/fssync/ssh.py
U zope.app.fssync/trunk/src/zope/app/fssync/syncer.py
A zope.app.fssync/trunk/src/zope/app/fssync/tests/ssh.txt
A zope.app.fssync/trunk/src/zope/app/fssync/tests/test_ssh.py
-=-
Property changes on: zope.app.fssync/trunk
___________________________________________________________________
Added: svn:ignore
+ bin
build
develop-eggs
dist
eggs
parts
.installed.cfg
Copied: zope.app.fssync/trunk/CHANGES.txt (from rev 97911, zope.app.fssync/branches/jim-hack/CHANGES.txt)
===================================================================
--- zope.app.fssync/trunk/CHANGES.txt (rev 0)
+++ zope.app.fssync/trunk/CHANGES.txt 2009-03-11 20:04:22 UTC (rev 97912)
@@ -0,0 +1,18 @@
+Changes
+=======
+
+3.5
+---
+
+* Added -v --verbose switches to zsync status command. Verbose is off
+ by default.
+
+* Added support for avoiding conflicts after commiting metadata files.
+
+* Added 'resolved' as an alias for the 'resolve' command.
+
+* Added a 'merge' command. It allows merging changes from one checkout
+ to another.
+
+* Added ssh network transport. The client can now use zsync+ssh://
+ urls to communicate with the server.
Modified: zope.app.fssync/trunk/README.txt
===================================================================
--- zope.app.fssync/trunk/README.txt 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/README.txt 2009-03-11 20:04:22 UTC (rev 97912)
@@ -1,41 +1,76 @@
-******
-FSSync
-******
+File Synchronization for Zope3
+==============================
.. contents::
-The FSSync project (zope.app.fssync) provides support for filesystem
+The FSSync project (zope.app.fssync) provides support for filesystem
synchronization of Zope3 content that resides in a ZODB. This package defines
-a Web-based API with basic support for some standard zope.app content
+a Web-based API with basic support for some standard zope.app content
types and the standard security policy.
-This project is build on top of the more general zope.fssync package which
-provides object serialization and deserialization tools. If you need a pure
-Python API which is independent of the ZODB and the Zope3 security machinery you
-should look at zope.fssync.
+This project is build on top of the more general zope.fssync package
+which provides object serialization and deserialization tools. If you
+need a pure Python API which is independent of the ZODB and the Zope3
+security machinery you should look at zope.fssync.
-FSSync includes a command line client that resembles svn or cvs. Type
+FSSync includes a command line client that resembles svn or cvs. Type::
bin/zsync help
-
-for available commands and further information. If you want to see the zsync
-client in action you can run the demo application:
- bin/demo -fg
+for available commands and further information. If you want to see the zsync
+client in action you can run the demo application::
-Open http://localhost:8080/manage in your browser and login with ``zsync`` as your
-username and password. Add a ``demo`` folder with some files via the ZMI.
-After that run the command line client for an initial checkout:
+ bin/demo start
+Open http://localhost:8080/manage in your browser and login with
+``zsync`` as your username and password. Add a ``demo`` folder with
+some files via the ZMI. After that run the command line client for an
+initial checkout::
+
bin/zsync checkout http://zsync:zsync@localhost:8080/demo ./parts/checkout
-Edit one of the files in the checkout directory and commit the changes:
+Edit one of the files in the checkout directory and commit the changes::
bin/zsync commit ./parts/checkout
-
+
The modified file should now be available on the server.
+SSH
+---
+
+Zsync now supports communication over ssh in addition to http. ssh
+urls look like::
+
+ zsync+ssh://user:passwd@host:port/path
+
+The zsync protocol is the same as over HTTP, it simply is sent via
+ssh.
+
+On the server side, the ssh server can check public keys to enforce
+security (though the demo server doesn't bother), and is responsible
+for passing the zsync request to zope and returning the response over
+ssh.
+
+There is an example ssh server in src/zope/app/fssync/demo_server.py
+To use it, first make sure that zope is running. Then start the
+server::
+
+ sudo bin/demo-ssh-server
+
+This starts a demo ssh server on port 2200. The server must be run as
+root in order to read the ssh host keys.
+
+In another terminal use the zsync client to connect to it::
+
+ bin/zsync co zsync+ssh://zsync:zsync@localhost:2200/demo parts/co2
+
+This checks out the demo folder into the parts/co2 folder.
+
+You should be able to work in the check out normally. Zsync will use
+ssh to communicate, but will otherwise act normally.
+
+
Extending zsync
---------------
@@ -47,10 +82,10 @@
entry-points = zsync=zope.app.fssync.main:main
If you want to use zope.app.fssync in your own application you propably have
-to define application specific serializers and deserializers. See
+to define application specific serializers and deserializers. See::
zope/app/fssync/fssync.txt
-
+
for further documentation. You probably also need your own zsync script with
additional dependencies. Simply add the necessary eggs to the corresponding
buildout snippet of your project.
Modified: zope.app.fssync/trunk/buildout.cfg
===================================================================
--- zope.app.fssync/trunk/buildout.cfg 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/buildout.cfg 2009-03-11 20:04:22 UTC (rev 97912)
@@ -1,7 +1,16 @@
[buildout]
develop = .
-parts = demo test zsync
+parts = demo test zsync zsync-demo-server
+index = http://pypi.python.org/simple
+extends = http://download.zope.org/zope3.4/3.4.0/versions.cfg
+versions = versions
+allow-picked-versions = false
+use-dependency-links = false
+[versions]
+paramiko = 1.7.4
+pycrypto = 2.0.1
+
[zope3]
location = .
@@ -23,7 +32,12 @@
[zsync]
recipe = zc.recipe.egg
eggs = zope.app.fssync
-entry-points = zsync=zope.app.fssync.main:main
+entry-points = zsync=zope.app.fssync.main:main
[database]
recipe = zc.recipe.filestorage
+
+[zsync-demo-server]
+recipe = zc.recipe.egg
+eggs = zope.app.fssync
+entry-points = demo-ssh-server=zope.app.fssync.demo_server:main
Modified: zope.app.fssync/trunk/setup.py
===================================================================
--- zope.app.fssync/trunk/setup.py 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/setup.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -2,21 +2,38 @@
from setuptools import setup, find_packages, Extension
+def read(*rnames):
+ return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
+
setup(name='zope.app.fssync',
- version = '3.4.0b1',
- url='http://svn.zope.org/zope.app.fssync',
- license='ZPL 2.1',
- description='Zope app fssync',
- author='Zope Corporation and Contributors',
- author_email='zope3-dev at zope.org',
- long_description="Filesystem synchronization for Zope 3 Applications.",
-
- packages=find_packages('src'),
- package_dir = {'': 'src'},
-
- namespace_packages=['zope',],
- tests_require = ['zope.testing'],
- extras_require = dict(
+ version = '3.5',
+ url = 'http://pypi.python.org/pypi/zope.app.fssync',
+ license = 'ZPL 2.1',
+ description = "Filesystem synchronization utility for Zope 3.",
+ author= 'Zope Corporation and Contributors',
+ author_email= 'zope3-dev at zope.org',
+ long_description = (read('README.txt')
+ + '\n\n' +
+ read('CHANGES.txt')
+ ),
+ packages=find_packages('src'),
+ package_dir = {'': 'src'},
+ namespace_packages=['zope', 'zope.app'],
+
+ keywords = "zope3 serialization synchronization",
+ classifiers = [
+ 'Development Status :: 5 - Production/Stable',
+ 'Environment :: Web Environment',
+ 'Intended Audience :: Developers',
+ 'License :: OSI Approved :: Zope Public License',
+ 'Programming Language :: Python',
+ 'Natural Language :: English',
+ 'Operating System :: OS Independent',
+ 'Topic :: Internet :: WWW/HTTP',
+ 'Framework :: Zope3'],
+
+ tests_require = ['zope.testing'],
+ extras_require = dict(
app = ['zope.app.appsetup',
'zope.app.authentication',
'zope.app.component',
@@ -31,25 +48,27 @@
'zope.app.wsgi',
]
),
- install_requires=['setuptools',
- 'zope.dublincore',
- 'zope.fssync',
- 'zope.interface',
- 'zope.proxy',
- 'zope.testbrowser',
- 'zope.traversing',
- 'zope.xmlpickle',
- 'zope.app.catalog',
- 'zope.app.component',
- 'zope.app.dtmlpage',
- 'zope.app.file',
- 'zope.app.folder',
- 'zope.app.module',
- 'zope.app.securitypolicy',
- 'zope.app.zcmlfiles',
- 'zope.app.zptpage'
- ],
- include_package_data = True,
-
- zip_safe = False,
- )
+ install_requires=['setuptools',
+ 'paramiko',
+ 'zope.dublincore',
+ 'zope.fssync >= 3.5',
+ 'zope.i18nmessageid',
+ 'zope.interface',
+ 'zope.proxy',
+ 'zope.testbrowser',
+ 'zope.traversing',
+ 'zope.xmlpickle',
+ 'zope.app.catalog',
+ 'zope.app.component',
+ 'zope.app.dtmlpage',
+ 'zope.app.file',
+ 'zope.app.folder',
+ 'zope.app.module',
+ 'zope.app.securitypolicy',
+ 'zope.app.zcmlfiles',
+ 'zope.app.zptpage'
+ ],
+ include_package_data = True,
+
+ zip_safe = False,
+ )
Modified: zope.app.fssync/trunk/src/zope/app/fssync/browser/__init__.py
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/browser/__init__.py 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/browser/__init__.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -29,7 +29,7 @@
from zope.fssync.snarf import Snarfer, Unsnarfer
from zope.fssync.metadata import Metadata
from zope.app.fssync import syncer
-from zope.app.i18n import ZopeMessageFactory as _
+from zope.i18nmessageid import ZopeMessageFactory as _
from zope.fssync import task
from zope.fssync import repository
Copied: zope.app.fssync/trunk/src/zope/app/fssync/demo_server.py (from rev 97911, zope.app.fssync/branches/jim-hack/src/zope/app/fssync/demo_server.py)
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/demo_server.py (rev 0)
+++ zope.app.fssync/trunk/src/zope/app/fssync/demo_server.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -0,0 +1,100 @@
+##############################################################################
+#
+# Copyright (c) 2009 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+import httplib
+import logging
+import socket
+
+
+import paramiko
+
+logging.basicConfig(level=logging.DEBUG)
+
+
+class Server(paramiko.ServerInterface):
+ """
+ Demo server configuration
+ """
+ def check_channel_request(self, kind, chanid):
+ if kind == 'session':
+ return paramiko.OPEN_SUCCEEDED
+ return paramiko.OPEN_FAILED_ADMINISTRATIVELY_PROHIBITED
+
+ def check_auth_publickey(self, username, key):
+ # All public keys are accepted for the demo, in a real server
+ # you would probably check public keys against keys of
+ # authorized users.
+ return paramiko.AUTH_SUCCESSFUL
+
+ def get_allowed_auths(self, username):
+ return 'publickey'
+
+
+class ZsyncHandler(paramiko.SubsystemHandler):
+ """
+ An example handler that forwards request to a zope server over
+ HTTP. A real server would probably communicate a different way.
+ """
+ def parse_request(self, channel, transport):
+ f = channel.makefile('r')
+ line = f.readline().strip('\r\n')
+ command, path = line.split(' ', 1)
+ command = command.strip()
+ path = path.strip()
+ headers = {}
+ while transport.is_active:
+ line = f.readline().strip('\r\n')
+ if not line:
+ break
+ key, value = line.split(':', 1)
+ headers[key.strip().lower()] = value.strip()
+ body = ''
+ length = int(headers.get('content-length', 0))
+ if length:
+ body = f.read(length)
+ return command, path, headers, body
+
+ def start_subsystem(self, name, transport, channel):
+ command, path, headers, body = self.parse_request(channel, transport)
+ connection = httplib.HTTPConnection('localhost', 8080)
+ if body:
+ connection.request(command, path, body=body, headers=headers)
+ else:
+ connection.request(command, path, headers=headers)
+ response = connection.getresponse()
+ channel.send('HTTP/1.0 %s %s\r\n' % (
+ response.status, response.reason))
+ for name, value in response.getheaders():
+ channel.send('%s: %s\r\n' % (name, value))
+ channel.send('\r\n')
+ body = response.read()
+ channel.sendall(body)
+
+
+def main(port=2200):
+ # read host keys
+ host_key = paramiko.RSAKey(filename = '/etc/ssh/ssh_host_rsa_key')
+
+ # start ssh server, install zsync handler
+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+ sock.settimeout(None)
+ sock.bind(('', port))
+ sock.listen(100)
+ while True:
+ client, addr = sock.accept()
+ client.settimeout(None)
+ t = paramiko.Transport(client)
+ t.add_server_key(host_key)
+ t.set_subsystem_handler('zsync', ZsyncHandler)
+ t.start_server(server = Server())
Modified: zope.app.fssync/trunk/src/zope/app/fssync/fssync.py
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/fssync.py 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/fssync.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -2,14 +2,14 @@
#
# Copyright (c) 2003 Zope Corporation and Contributors.
# All Rights Reserved.
-#
+#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
-#
+#
##############################################################################
"""Highest-level classes to support filesystem synchronization:
@@ -32,6 +32,7 @@
from StringIO import StringIO
+import os.path
from os.path import exists, isfile, isdir
from os.path import dirname, basename, split, join
from os.path import realpath, normcase, normpath
@@ -42,6 +43,8 @@
from zope.fssync import fsutil
from zope.fssync.snarf import Snarfer, Unsnarfer
from zope.app.fssync.passwd import PasswordManager
+from zope.app.fssync.ssh import SSHConnection
+import zope.app.fssync.merge
if sys.platform[:3].lower() == "win":
DEV_NULL = r".\nul"
@@ -136,8 +139,9 @@
rooturl = roottype = rootpath = user_passwd = host_port = None
else:
roottype, rest = urllib.splittype(rooturl)
- if roottype not in ("http", "https"):
- raise Error("root url must be 'http' or 'https'", rooturl)
+ if roottype not in ("http", "https", "zsync+ssh"):
+ raise Error("root url must be 'http', 'https', or 'zsync+ssh'",
+ rooturl)
if roottype == "https" and not hasattr(httplib, "HTTPS"):
raise Error("https not supported by this Python build")
netloc, rootpath = urllib.splithost(rest)
@@ -207,13 +211,15 @@
assert self.rooturl
if not path.endswith("/"):
path += "/"
- path = urllib.quote(path)
+ path = urllib.quote(path)
path += view
if self.roottype == "https":
conn = httplib.HTTPSConnection(self.host_port)
+ elif self.roottype == "zsync+ssh":
+ conn = SSHConnection(self.host_port, self.user_passwd)
else:
conn = httplib.HTTPConnection(self.host_port)
-
+
if datasource is None:
conn.putrequest("GET", path)
else:
@@ -228,7 +234,7 @@
tmp = tempfile.TemporaryFile('w+b')
datasource(tmp)
conn.putheader("Content-Length", str(tmp.tell()))
-
+
if self.user_passwd:
if ":" not in self.user_passwd:
auth = self.getToken(self.roottype,
@@ -241,7 +247,7 @@
conn.putheader("Connection", "close")
conn.endheaders()
if datasource is not None:
- #XXX If chunking works again, replace the following lines with
+ #XXX If chunking works again, replace the following lines with
# datasource(PretendStream(conn))
# conn.send("0\r\n\r\n")
tmp.seek(0)
@@ -249,8 +255,8 @@
while data:
conn.send(data)
data = tmp.read(1<<16)
- tmp.close()
-
+ tmp.close()
+
response = conn.getresponse()
if response.status != 200:
raise Error("HTTP error %s (%s); error document:\n%s",
@@ -309,7 +315,8 @@
class FSSync(object):
- def __init__(self, metadata=None, network=None, rooturl=None):
+ def __init__(self, metadata=None, network=None, rooturl=None,
+ overwrite_local=False):
if metadata is None:
metadata = Metadata()
if network is None:
@@ -317,7 +324,8 @@
self.metadata = metadata
self.network = network
self.network.setrooturl(rooturl)
- self.fsmerger = FSMerger(self.metadata, self.reporter)
+ self.fsmerger = FSMerger(self.metadata, self.reporter,
+ overwrite_local)
def login(self, url=None, user=None):
scheme, host_port, user = self.get_login_info(url, user)
@@ -459,6 +467,40 @@
finally:
fp.close()
+ def merge(self, args):
+ source = args[0]
+ if len(args) == 1:
+ target = os.curdir
+ else:
+ target = args[1]
+
+ # make sure that we're merging from compatible directories
+ if not self.metadata.getentry(target):
+ names = self.metadata.getnames(target)
+ if len(names) == 1:
+ target = join(target, names[0])
+ target_entry = self.metadata.getentry(target)
+ if not target_entry:
+ print 'Target must be a fssync checkout directory'
+ return
+ if not self.metadata.getentry(source):
+ names = self.metadata.getnames(source)
+ if len(names) == 1:
+ source = join(source, names[0])
+ source_entry = self.metadata.getentry(source)
+ if not source_entry:
+ print 'Source must be a fssync checkout directory'
+ return
+ if source_entry[u'id'] != target_entry[u'id']:
+ print 'Cannot merge from %s to %s' % (source_entry[u'id'],
+ target_entry[u'id'])
+ return
+
+ zope.app.fssync.merge.merge(os.path.abspath(source),
+ os.path.abspath(target), self)
+ print 'All done.'
+
+
def merge_snarffile(self, fp, localdir, tail):
uns = Unsnarfer(fp)
tmpdir = tempfile.mktemp()
@@ -529,7 +571,7 @@
def reporter(self, msg):
if msg[0] not in "/*":
print msg.encode('utf-8') # uo: is encode needed here?
-
+
def diff(self, target, mode=1, diffopts="", need_original=True):
assert mode == 1, "modes 2 and 3 are not yet supported"
entry = self.metadata.getentry(target)
@@ -669,7 +711,7 @@
self.metadata.flush()
print "R", path
- def status(self, target, descend_only=False):
+ def status(self, target, descend_only=False, verbose=True):
entry = self.metadata.getentry(target)
flag = entry.get("flag")
if isfile(target):
@@ -684,7 +726,8 @@
original = fsutil.getoriginal(target)
if isfile(original):
if filecmp.cmp(target, original):
- print "=", target
+ if verbose:
+ print "=", target
else:
print "M", target
else:
@@ -698,7 +741,7 @@
print "A", pname
elif flag == "removed":
print "R(reborn)", pname
- else:
+ elif verbose:
print "/", pname
if entry:
# Recurse down the directory
@@ -713,7 +756,8 @@
ncnames = namesdir.keys()
ncnames.sort()
for ncname in ncnames:
- self.status(join(target, namesdir[ncname]))
+ self.status(join(target, namesdir[ncname]),
+ verbose=verbose)
elif exists(target):
if not entry:
if not self.fsmerger.ignore(target):
@@ -733,10 +777,10 @@
print "lost", target
annotations = fsutil.getannotations(target)
if isdir(annotations):
- self.status(annotations, True)
+ self.status(annotations, True, verbose=verbose)
extra = fsutil.getextra(target)
if isdir(extra):
- self.status(extra, True)
+ self.status(extra, True, verbose=verbose)
def quote(s):
"""Helper to put quotes around arguments passed to shell if necessary."""
Modified: zope.app.fssync/trunk/src/zope/app/fssync/fssync.txt
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/fssync.txt 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/fssync.txt 2009-03-11 20:04:22 UTC (rev 97912)
@@ -2,10 +2,10 @@
============
The fssync package allows users to download objects from a Zope3 server
-to the local disk, edit the objects offline and synchronize the
+to the local disk, edit the objects offline and synchronize the
modifications with the server later on.
-Let's start with some basic infrastructure on the server side. We
+Let's start with some basic infrastructure on the server side. We
assume that a folder with some content already exists:
>>> root = getRootFolder()
@@ -30,13 +30,13 @@
Serialization format
--------------------
-On the server side everything must be registered in a manner that we
-are allowed to access the serialized data (see registration.txt for
-details). The serialized content is delivered in a Zope3 specific
+On the server side everything must be registered in a manner that we
+are allowed to access the serialized data (see registration.txt for
+details). The serialized content is delivered in a Zope3 specific
SNARF archive.
-SNARF (Simple New ARchival Format) is a very simple format that
-basically puts one file after another. Here we download it by calling
-the @@toFS.snarf view to give an impression of the internal structure
+SNARF (Simple New ARchival Format) is a very simple format that
+basically puts one file after another. Here we download it by calling
+the @@toFS.snarf view to give an impression of the internal structure
of this format:
>>> headers = {'Authorization':'Basic globalmgr:globalmgrpw'}
@@ -134,22 +134,21 @@
A text file00000017 test/file2.txt
Another text file
-
-Note that the main content is directly serialized whereas extra
+Note that the main content is directly serialized whereas extra
attributes and metadata are pickled in an XML format.
Initial Checkout
----------------
-We perform an initial checkout to see what happens. We mimic the
-command line syntax
+We perform an initial checkout to see what happens. We mimic the
+command line syntax
zsync checkout http://user:password@host:port/path targetdir
-by using the corresponding FSSync command object. (The zsync script
-can be found in this directory. Type ``zsync help`` for a list of
-available commands).
+by using the corresponding FSSync command object. (The zsync script
+can be found in this directory. Type ``zsync help`` for a list of
+available commands).
The FSSync object must be initialized with all relevant connection data.
For the sake of this doctest we need also a special network instance:
@@ -176,11 +175,12 @@
U .../@@Zope/Annotations/test/zope.app.dublincore.ZopeDublinCore
All done.
-The printout shows all new directories and updated files. As you can see,
-the file content is directly mapped onto the filesystem whereas extra data
-and metadata are stored in special @@Zope
+The printout shows all new directories and updated files. As you can see,
+the file content is directly mapped onto the filesystem whereas extra data
+and metadata are stored in special @@Zope
directories.
+
Local Modifications
-------------------
@@ -199,17 +199,22 @@
M .../test/file1.txt
= .../test/file2.txt
-If we want to add a file to the repository we must update the local
+You can also turn off verbose mode to have it only list interesting lines.
+
+ >>> zsync.status(localdir, verbose=False)
+ M .../test/file1.txt
+
+If we want to add a file to the repository we must update the local
list of entries by calling the add command explicitely:
>>> newlocalfile = os.path.join(localdir, 'file3.txt')
>>> fp = open(newlocalfile, 'w')
>>> fp.write('A new local text file')
>>> fp.close()
-
+
>>> zsync.add(newlocalfile)
A .../test/file3.txt
-
+
>>> zsync.status(localdir)
/ .../test/
M .../test/file1.txt
@@ -221,12 +226,12 @@
-----------------------
Before we commit our local modifications we should check whether our
-local repository is still up to date. Let's say that by a coincidence
+local repository is still up to date. Let's say that by a coincidence
someone else edited the same file on the server:
>>> serverfile1.data = 'Ooops'
>>> zope.event.notify(ObjectModifiedEvent(serverfile1))
-
+
>>> zsync.commit(localdir)
Traceback (most recent call last):
...
@@ -234,7 +239,7 @@
test/file1.txt
test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
-We must update the local files and resolve all conflicts before
+We must update the local files and resolve all conflicts before
we can proceed:
>>> zsync.update(localdir)
@@ -261,31 +266,31 @@
Now we can commit our work and have a look at the resulting events:
>>> def traceEvent(event):
- ... print event.__class__.__name__,
+ ... print event.__class__.__name__,
... print getattr(event.object, '__name__', ''),
... descriptions = getattr(event, 'descriptions', None)
... if descriptions is not None:
... for desc in descriptions:
... print desc.__class__.__name__,
... print ''
-
+
>>> zope.event.subscribers.append(traceEvent)
>>> import time
>>> time.sleep(0.1)
>>> zsync.commit(localdir)
- BeforeTraverseEvent None
- BeforeTraverseEvent test
- BeforeTraverseEvent fromFS.snarf
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent file1.txt ObjectSynchronized
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent ObjectSynchronized
- ObjectCreatedEvent
- ObjectAddedEvent file3.txt
- ContainerModifiedEvent test
- ObjectModifiedEvent ObjectSynchronized
- EndRequestEvent run
+ BeforeTraverseEvent None
+ BeforeTraverseEvent test
+ BeforeTraverseEvent fromFS.snarf
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent file1.txt ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectCreatedEvent
+ ObjectAddedEvent file3.txt
+ ContainerModifiedEvent test
+ ObjectModifiedEvent ObjectSynchronized
+ EndRequestEvent run
U .../test/file1.txt
U .../test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
U .../test/file3.txt
@@ -295,7 +300,6 @@
U .../test/@@Zope/Annotations/file3.txt/zope.app.dublincore.ZopeDublinCore
All done.
-
Let's check whether the server objects have been updated accordingly:
>>> serverfile1.data
@@ -309,7 +313,7 @@
Checkin
-------
-If we want to import (or reimport) the data into a content space
+If we want to import (or reimport) the data into a content space
we can use the checkin command:
>>> del root[u'test']
@@ -317,31 +321,31 @@
ContainerModifiedEvent None
>>> zsync.checkin(localdir)
- BeforeTraverseEvent None
- BeforeTraverseEvent checkin.snarf
- ObjectCreatedEvent None
- ObjectAddedEvent test
- ContainerModifiedEvent None
- ObjectCreatedEvent
- ObjectAddedEvent file1.txt
- ContainerModifiedEvent test
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent file1.txt Attributes
- ObjectCreatedEvent
- ObjectAddedEvent file2.txt
- ContainerModifiedEvent test
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent file2.txt Attributes
- ObjectCreatedEvent
- ObjectAddedEvent file3.txt
- ContainerModifiedEvent test
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent ObjectSynchronized
- ObjectModifiedEvent file3.txt Attributes
- ObjectModifiedEvent ObjectSynchronized
- EndRequestEvent run
+ BeforeTraverseEvent None
+ BeforeTraverseEvent checkin.snarf
+ ObjectCreatedEvent None
+ ObjectAddedEvent test
+ ContainerModifiedEvent None
+ ObjectCreatedEvent
+ ObjectAddedEvent file1.txt
+ ContainerModifiedEvent test
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent file1.txt Attributes
+ ObjectCreatedEvent
+ ObjectAddedEvent file2.txt
+ ContainerModifiedEvent test
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent file2.txt Attributes
+ ObjectCreatedEvent
+ ObjectAddedEvent file3.txt
+ ContainerModifiedEvent test
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent file3.txt Attributes
+ ObjectModifiedEvent ObjectSynchronized
+ EndRequestEvent run
>>> serverfolder = root[u'test']
>>> sorted(serverfolder.keys())
@@ -351,8 +355,8 @@
A resolved conflict
>>> serverfile1.getSize() == len(serverfile1.data)
True
-
-We need to make sure that the top-level name doesn't already exist,
+
+We need to make sure that the top-level name doesn't already exist,
or existing data can get screwed:
>>> zsync.checkin(localdir)
@@ -361,9 +365,117 @@
SynchronizationError: object already exists 'test'
+Let's test changing metadata.
+First let's examine an existing metadata file.
+
+ >>> path = (os.path.join(localdir, '@@Zope', 'Extra', 'file1.txt',
+ ... 'contentType'))
+ >>> open(path).read()
+ '<?xml version="1.0" encoding="utf-8" ?>\n<pickle> <string>text/plain</string> </pickle>\n'
+
+Now let's change it.
+
+ >>> f = open(path, 'w')
+ >>> f.write('<?xml version="1.0" encoding="utf-8" ?>\n<pickle>\n')
+ >>> f.write('<string>text/html</string>\n</pickle>\n')
+ >>> f.close()
+
+Now we commit our changes.
+
+ >>> zsync.update(localdir)
+ BeforeTraverseEvent None
+ BeforeTraverseEvent test
+ BeforeTraverseEvent toFS.snarf
+ EndRequestEvent show
+ M .../test/@@Zope/Extra/file1.txt/contentType
+ U .../test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
+ U .../test/@@Zope/Annotations/file2.txt/zope.app.dublincore.ZopeDublinCore
+ U .../test/@@Zope/Annotations/file3.txt/zope.app.dublincore.ZopeDublinCore
+ All done.
+
+ >>> zsync.commit(localdir)
+ BeforeTraverseEvent None
+ BeforeTraverseEvent test
+ BeforeTraverseEvent fromFS.snarf
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent file1.txt Attributes
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ EndRequestEvent run
+ C .../test/@@Zope/Extra/file1.txt/contentType
+ U .../test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
+ All done.
+
+The problem is that we formatted the XML slightly differently than the
+synchronizer does. Thus the client thinks there's a conflict after the
+commit. The solution is to set overwrite_local to True on the FSSync's
+merger object. (This is normally done in the FSSync constructor, but
+we'll do it manually in this test.)
+
+Let's return things to how they were.
+
+ >>> zsync.revert(path)
+ Reverted .../test/@@Zope/Extra/file1.txt/contentType
+
+ >>> serverfile1.contentType = 'text/plain'
+ >>> zsync.update(localdir)
+ BeforeTraverseEvent None
+ BeforeTraverseEvent test
+ BeforeTraverseEvent toFS.snarf
+ EndRequestEvent show
+ U .../test/@@Zope/Extra/file1.txt/contentType
+ All done.
+
+Make our metadata change again.
+
+ >>> f = open(path, 'w')
+ >>> f.write('<?xml version="1.0" encoding="utf-8" ?>\n<pickle>\n')
+ >>> f.write('<string>text/html</string>\n</pickle>\n')
+ >>> f.close()
+ >>> zsync.update(localdir)
+ BeforeTraverseEvent None
+ BeforeTraverseEvent test
+ BeforeTraverseEvent toFS.snarf
+ EndRequestEvent show
+ M .../test/@@Zope/Extra/file1.txt/contentType
+ All done.
+
+Now we commit again, but this time using overwrite_local.
+
+ >>> zsync.fsmerger.overwrite_local = True
+ >>> zsync.commit(localdir)
+ BeforeTraverseEvent None
+ BeforeTraverseEvent test
+ BeforeTraverseEvent fromFS.snarf
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent file1.txt Attributes
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ ObjectModifiedEvent ObjectSynchronized
+ EndRequestEvent run
+ U .../test/@@Zope/Extra/file1.txt/contentType
+ U .../test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
+ All done.
+
+Let's confirm that the change was made.
+
+ >>> serverfile1.contentType
+ 'text/html'
+
+Also, note that our metadata file was overwritten with the server verion.
+
+ >>> open(path).read()
+ '<?xml version="1.0" encoding="utf-8" ?>/n<pickle> <string>text/html</string> </pickle>/n'
+
Clean up
--------
>>> zope.event.subscribers.remove(traceEvent)
-
Modified: zope.app.fssync/trunk/src/zope/app/fssync/ftesting.zcml
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/ftesting.zcml 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/ftesting.zcml 2009-03-11 20:04:22 UTC (rev 97912)
@@ -20,7 +20,7 @@
<include package="zope.app.fssync"/>
<securityPolicy
- component="zope.app.securitypolicy.zopepolicy.ZopeSecurityPolicy" />
+ component="zope.securitypolicy.zopepolicy.ZopeSecurityPolicy" />
<role id="zope.Anonymous" title="Everybody"
description="All users have this role implicitly" />
Modified: zope.app.fssync/trunk/src/zope/app/fssync/ftests.py
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/ftests.py 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/ftests.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -31,6 +31,7 @@
from zope.app.fssync.testing import TestNetwork
checkoutdir = tempfile.mkdtemp(prefix='checkoutdir')
+checkoutdir2 = tempfile.mkdtemp(prefix='checkoutdir2')
checker = renormalizing.RENormalizing([
(re.compile(r"\\"), r"/"),
@@ -40,10 +41,13 @@
module.setUp(test, 'zope.app.fssync.fssync_txt')
if not os.path.exists(checkoutdir):
os.mkdir(checkoutdir)
+ if not os.path.exists(checkoutdir2):
+ os.mkdir(checkoutdir2)
def tearDown(test):
module.tearDown(test, 'zope.app.fssync.fssync_txt')
shutil.rmtree(checkoutdir)
+ shutil.rmtree(checkoutdir2)
def cleanUpTree(dir):
if os.path.exists(dir):
@@ -51,11 +55,12 @@
os.mkdir(dir)
def test_suite():
-
+
globs = {'os': os,
'zope':zope,
'pprint': doctestunit.pprint,
'checkoutdir':checkoutdir,
+ 'checkoutdir2': checkoutdir2,
'cleanUpTree': cleanUpTree,
'PublisherConnection': PublisherConnection,
'TestNetwork': TestNetwork,
@@ -63,14 +68,14 @@
suite = unittest.TestSuite()
- for file in 'fssync.txt', 'security.txt', 'fssite.txt':
+ for file in 'fssync.txt', 'security.txt', 'fssite.txt', 'merge.txt':
test = functional.FunctionalDocFileSuite(file,
setUp=setUp, tearDown=tearDown,
globs=globs, checker=checker,
optionflags=doctest.NORMALIZE_WHITESPACE+doctest.ELLIPSIS)
test.layer = AppFSSyncLayer
suite.addTest(test)
-
+
if sys.platform != 'win32':
test = functional.FunctionalDocFileSuite('fsmerge.txt',
setUp=setUp, tearDown=tearDown,
Modified: zope.app.fssync/trunk/src/zope/app/fssync/main.py
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/main.py 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/main.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -3,14 +3,14 @@
#
# Copyright (c) 2003 Zope Corporation and Contributors.
# All Rights Reserved.
-#
+#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
-#
+#
##############################################################################
"""Filesystem synchronization utility for Zope 3.
@@ -28,7 +28,7 @@
%(program)s remove [options] TARGET ...
%(program)s resolve PATH ...
%(program)s revert PATH ...
-%(program)s status [TARGET ...]
+%(program)s status [options] [TARGET ...]
%(program)s update [TARGET ...]
``%(program)s help'' prints the global help (this message)
@@ -127,7 +127,7 @@
for o, a in opts:
if o in ("-r", "--raise-on-conflicts"):
raise_on_conflicts = True
- fs = FSSync()
+ fs = FSSync(overwrite_local=True)
fs.multiple(args, fs.commit, message, raise_on_conflicts)
def update(opts, args):
@@ -254,13 +254,20 @@
fs.multiple(args, fs.diff, mode, diffopts, need_original)
def status(opts, args):
- """%(program)s status [TARGET ...]
+ """%(program)s status [-v] [--verbose] [TARGET ...]
Print brief (local) status for each target, without changing any
files or contacting the Zope server.
+
+ If the -v or --verbose switches are provided, prints a complete
+ list of local files regardless of their status.
"""
+ verbose = False
+ for o, a in opts:
+ if o in ('-v', '--verbose'):
+ verbose = True
fs = FSSync()
- fs.multiple(args, fs.status)
+ fs.multiple(args, fs.status, False, verbose)
def checkin(opts, args):
"""%(program)s checkin [-m message] URL [TARGETDIR]
@@ -287,7 +294,7 @@
raise Usage("checkin requires at most one TARGETDIR argument")
else:
target = os.curdir
- fs = FSSync(rooturl=rooturl)
+ fs = FSSync(rooturl=rooturl, overwrite_local=True)
fs.checkin(target, message)
def login(opts, args):
@@ -352,6 +359,19 @@
fs = FSSync()
fs.multiple(args, fs.revert)
+def merge(opts, args):
+ """%(program)s merge [TARGETDIR] SOURCEDIR
+
+ Merge changes from one sandbox directory to another. If two
+ directories are specified then the first one is the target and the
+ second is the source. If only one directory is specified, then the
+ target directory is assumed to the be current sandbox.
+ """
+ if len(args) not in (1,2):
+ raise Usage('Merge requires one or two arguments')
+ fs = FSSync()
+ fs.merge(args)
+
def extract_message(opts, cmd):
L = []
message = None
@@ -387,10 +407,11 @@
(diff, "di", "bBcC:iNuU:", "brief context= unified="),
(login, "", "u:", "user="),
(logout, "", "u:", "user="),
+ (merge, "", "", ""),
(mkdir, "", "", ""),
(remove, "del delete rm", "", ""),
- (resolve, "", "", ""),
+ (resolve, "resolved","", ""),
(revert, "", "", ""),
- (status, "stat st", "", ""),
+ (status, "stat st", "v", "verbose"),
(update, "up", "", ""),
]
Copied: zope.app.fssync/trunk/src/zope/app/fssync/merge.py (from rev 97911, zope.app.fssync/branches/jim-hack/src/zope/app/fssync/merge.py)
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/merge.py (rev 0)
+++ zope.app.fssync/trunk/src/zope/app/fssync/merge.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -0,0 +1,144 @@
+##############################################################################
+#
+# Copyright (c) 2009 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""
+Merge from one fssync repository to another
+
+This differs from merging from Zope in that changes from the source
+are not considered changes to the orginal.
+"""
+import os
+import os.path
+import shutil
+
+import zope.fssync.fsutil
+import zope.fssync.copier
+
+class MergeCopier(zope.fssync.copier.ObjectCopier):
+ """
+ Copies from a checkout to another checkout, but doesn't add sync
+ entries in the destination checkout, since they should already be
+ present.
+ """
+
+ def copy(self, source, target, children=True):
+ if os.path.isdir(source):
+ shutil.copymode(source, target)
+ self.addEntry(source, target)
+ else:
+ shutil.copy(source, target)
+ self.addEntry(source, target)
+
+ def _copyspecials(self, source, target, getwhat):
+ src = getwhat(source)
+ if os.path.isdir(src):
+ dst = getwhat(target)
+ zope.fssync.fsutil.ensuredir(dst)
+ copier = MergeCopier(self.sync)
+ for name in self.sync.metadata.getnames(src):
+ copier.copy(os.path.join(src, name), os.path.join(dst, name))
+ self.sync.metadata.flush()
+
+ def _syncadd(self, target, type, factory):
+ pass
+
+
+def same_type(path1, path2):
+ if (os.path.isfile(path1) == os.path.isfile(path2) or
+ os.path.isdir(path1) == os.path.isdir(path2)):
+ return True
+ return False
+
+def same_file(path1, path2):
+ chunk_size = 32768
+ f1 = open(path1)
+ f2 = open(path2)
+ while True:
+ s1 = f1.read(chunk_size)
+ s2 = f2.read(chunk_size)
+ if s1 != s2:
+ return False
+ if not s1:
+ break
+ f1.close()
+ f2.close()
+ return True
+
+def same_entries(path1, path2, metadata):
+ names1 = metadata.getnames(path1)
+ names2 = metadata.getnames(path2)
+ if names1 != names2:
+ return False
+ for name in names1:
+ f1 = os.path.join(path1, name)
+ f2 = os.path.join(path2, name)
+ if not same_file(f1, f2):
+ return False
+ return True
+
+def same_specials(path1, path2, metadata):
+ extra1 = zope.fssync.fsutil.getextra(path1)
+ extra2 = zope.fssync.fsutil.getextra(path2)
+ if os.path.exists(extra1) != os.path.exists(extra2):
+ return False
+ if os.path.exists(extra1) and os.path.exists(extra2):
+ if not same_entries(extra1, extra2, metadata):
+ return False
+
+ annotations1 = zope.fssync.fsutil.getannotations(path1)
+ annotations2 = zope.fssync.fsutil.getannotations(path2)
+ if os.path.exists(annotations1) != os.path.exists(annotations2):
+ return False
+ if os.path.exists(annotations1) and os.path.exists(annotations2):
+ if not same_entries(annotations1, annotations2, metadata):
+ return False
+
+ return True
+
+
+
+def merge(source, target, sync):
+ """
+ Merge difference from source into target. Treat differences as
+ local changes in target. Don't delete anything from target, only
+ add things from source. Only merge stuff from target if it is
+ actually in the repository (thus don't copy temp files, svn files,
+ etc.)
+ """
+ metadata = sync.metadata
+ object_copier = zope.fssync.copier.ObjectCopier(sync)
+ merge_copier = MergeCopier(sync)
+
+ for root, dirs, files in os.walk(source):
+ if '@@Zope' in dirs:
+ dirs.remove('@@Zope')
+ for filename in ([''] + files):
+ source_path = os.path.join(root, filename)
+ if metadata.getentry(source_path):
+ directory = root[len(source) + 1:]
+ target_path = os.path.join(target, directory, filename)
+ if not metadata.getentry(target_path):
+ object_copier.copy(source_path, target_path, False)
+ elif not same_type(source_path, target_path):
+ print 'C %s' % target_path
+ metadata.getentry(target_path)['conflict'] = True
+ metadata.flush()
+ elif os.path.isfile(source_path):
+ if (not same_file(source_path, target_path) or
+ not same_specials(source_path, target_path, metadata)):
+ print 'M %s' % target_path
+ merge_copier.copy(source_path, target_path, False)
+ elif os.path.isdir(source_path):
+ if not same_specials(source_path, target_path, metadata):
+ print 'M %s' % target_path
+ merge_copier.copy(source_path, target_path, False)
Copied: zope.app.fssync/trunk/src/zope/app/fssync/merge.txt (from rev 97911, zope.app.fssync/branches/jim-hack/src/zope/app/fssync/merge.txt)
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/merge.txt (rev 0)
+++ zope.app.fssync/trunk/src/zope/app/fssync/merge.txt 2009-03-11 20:04:22 UTC (rev 97912)
@@ -0,0 +1,173 @@
+Merge Command
+=============
+
+The merge command copies changes from one checkout to another. It only
+copies changes and additions; it doesn't copy deletions. Its purpose
+is to allow changes on one server to be moved to another via
+fssync. For example, it could be used to move changes from a
+development to a production instance.
+
+Let's begin by creating a server with some data.
+
+ >>> root = getRootFolder()
+ >>> from zope.app.folder import Folder
+ >>> from zope.lifecycleevent import ObjectCreatedEvent
+ >>> serverfolder = root[u'test'] = Folder()
+ >>> from zope.app.file import File
+ >>> serverfile1 = File('A\nB\nC', 'text/plain')
+ >>> zope.event.notify(ObjectCreatedEvent(serverfile1))
+ >>> serverfolder[u'file1.txt'] = serverfile1
+ >>> serverfile2 = File('A\nB\nC', 'text/plain')
+ >>> zope.event.notify(ObjectCreatedEvent(serverfile2))
+ >>> serverfolder[u'file2.txt'] = serverfile2
+
+Now let's create two checkouts.
+
+ >>> os.path.exists(checkoutdir)
+ True
+ >>> os.path.exists(checkoutdir2)
+ True
+
+Now let's do the checkouts.
+
+ >>> from zope.app.fssync.fssync import FSSync
+ >>> rooturl = 'http://globalmgr:globalmgrpw@localhost/test'
+ >>> network = TestNetwork(handle_errors=True)
+ >>> zsync = FSSync(network=network, rooturl=rooturl)
+
+ >>> zsync.checkout(checkoutdir)
+ N .../test/
+ U .../test/file1.txt
+ N .../test/@@Zope/Extra/file1.txt/
+ U .../test/@@Zope/Extra/file1.txt/contentType
+ N .../test/@@Zope/Annotations/file1.txt/
+ U .../test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
+ U .../test/file2.txt
+ N .../test/@@Zope/Extra/file2.txt/
+ U .../test/@@Zope/Extra/file2.txt/contentType
+ N .../test/@@Zope/Annotations/file2.txt/
+ U .../test/@@Zope/Annotations/file2.txt/zope.app.dublincore.ZopeDublinCore
+ N .../@@Zope/Annotations/test/
+ U .../@@Zope/Annotations/test/zope.app.dublincore.ZopeDublinCore
+ All done.
+
+ >>> zsync.checkout(checkoutdir2)
+ N .../test/
+ U .../test/file1.txt
+ N .../test/@@Zope/Extra/file1.txt/
+ U .../test/@@Zope/Extra/file1.txt/contentType
+ N .../test/@@Zope/Annotations/file1.txt/
+ U .../test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
+ U .../test/file2.txt
+ N .../test/@@Zope/Extra/file2.txt/
+ U .../test/@@Zope/Extra/file2.txt/contentType
+ N .../test/@@Zope/Annotations/file2.txt/
+ U .../test/@@Zope/Annotations/file2.txt/zope.app.dublincore.ZopeDublinCore
+ N .../@@Zope/Annotations/test/
+ U .../@@Zope/Annotations/test/zope.app.dublincore.ZopeDublinCore
+ All done.
+
+Now we'll make some changes to one of the checkouts.
+
+ >>> localfile1 = os.path.join(checkoutdir, 'test', 'file1.txt')
+ >>> fp = open(localfile1, 'w')
+ >>> fp.write('A modified text file')
+ >>> fp.close()
+
+An addition.
+
+ >>> localfile3 = os.path.join(checkoutdir, 'test', 'file3.txt')
+ >>> fp = open(localfile3, 'w')
+ >>> fp.write('A new text file')
+ >>> fp.close()
+ >>> zsync.add(localfile3)
+ A .../test/file3.txt
+
+A new directory.
+
+ >>> sitedir = os.path.join(checkoutdir, 'test', 'newfolder')
+ >>> os.mkdir(sitedir)
+ >>> zsync.add(sitedir)
+ A .../test/newfolder/
+
+Here's a deletion.
+
+ >>> os.unlink(os.path.join(checkoutdir, 'test', 'file2.txt'))
+ >>> zsync.remove(os.path.join(checkoutdir, 'test', 'file2.txt'))
+ R .../test/file2.txt
+
+Now we commit our changes.
+
+ >>> zsync.commit(os.path.join(checkoutdir, 'test'))
+ U .../test/file1.txt
+ U .../test/@@Zope/Annotations/file1.txt/zope.app.dublincore.ZopeDublinCore
+ D .../test/file2.txt
+ D .../test/@@Zope/Extra/file2.txt/contentType
+ D .../test/@@Zope/Extra/file2.txt/
+ D .../test/@@Zope/Annotations/file2.txt/zope.app.dublincore.ZopeDublinCore
+ D .../test/@@Zope/Annotations/file2.txt/
+ U .../test/file3.txt
+ N .../test/@@Zope/Extra/file3.txt/
+ U .../test/@@Zope/Extra/file3.txt/contentType
+ N .../test/@@Zope/Annotations/file3.txt/
+ U .../test/@@Zope/Annotations/file3.txt/zope.app.dublincore.ZopeDublinCore
+ U .../test/newfolder/
+ N .../test/@@Zope/Annotations/newfolder/
+ U .../test/@@Zope/Annotations/newfolder/zope.app.dublincore.ZopeDublinCore
+ All done.
+
+At this point the two checkouts are not in sync.
+
+ >>> f1 = open(os.path.join(checkoutdir, 'test', 'file1.txt'))
+ >>> f2 = open(os.path.join(checkoutdir2, 'test', 'file1.txt'))
+ >>> f1.read() == f2.read()
+ False
+
+ >>> f1.close()
+ >>> f2.close()
+
+Now we'll merge changes from one checkout to the other.
+
+ >>> zsync.merge((checkoutdir, checkoutdir2))
+ M .../test/file1.txt
+ A .../test/file3.txt
+ A .../test/newfolder/
+ All done.
+
+The status command reflects the local changes.
+
+ >>> zsync.status(os.path.join(checkoutdir2, 'test'))
+ / .../test/
+ M .../test/file1.txt
+ = .../test/file2.txt
+ A .../test/file3.txt
+ A .../test/newfolder/
+
+The change and the addition were picked up, but the deletion wasn't.
+
+Let's confirm that the changed and added files made it.
+
+ >>> f1 = open(os.path.join(checkoutdir, 'test', 'file1.txt'))
+ >>> f2 = open(os.path.join(checkoutdir2, 'test', 'file1.txt'))
+ >>> f1.read() == f2.read()
+ True
+
+ >>> f1.close()
+ >>> f2.close()
+
+ >>> f1 = open(os.path.join(checkoutdir, 'test', 'file3.txt'))
+ >>> f2 = open(os.path.join(checkoutdir2, 'test', 'file3.txt'))
+ >>> f1.read() == f2.read()
+ True
+
+ >>> f1.close()
+ >>> f2.close()
+
+ >>> os.path.isdir(os.path.join(checkoutdir2, 'test', 'newfolder'))
+ True
+
+Let's make sure that extras were also copied over.
+
+ >>> open(os.path.join(checkoutdir2, 'test', '@@Zope', 'Extra',
+ ... 'file3.txt', 'contentType')).read()
+ '<?xml version="1.0" encoding="utf-8" ?>\n<pickle> <string>text/plain</string> </pickle>\n'
Modified: zope.app.fssync/trunk/src/zope/app/fssync/passwd.py
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/passwd.py 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/passwd.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -22,6 +22,7 @@
from cStringIO import StringIO
+from zope.fssync import fsutil
DEFAULT_FILENAME = os.path.expanduser(os.path.join("~", ".zsyncpass"))
@@ -179,6 +180,8 @@
return _normalize_port(host_port, httplib.HTTP_PORT)
elif scheme == "https":
return _normalize_port(host_port, httplib.HTTPS_PORT)
+ elif scheme == "zsync+ssh":
+ return _normalize_port(host_port, None)
else:
raise fsutil.Error("unsupported URL scheme: %r" % scheme)
Modified: zope.app.fssync/trunk/src/zope/app/fssync/security.txt
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/security.txt 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/security.txt 2009-03-11 20:04:22 UTC (rev 97912)
@@ -123,7 +123,7 @@
>>> readonly.commit(localfile1)
Traceback (most recent call last):
...
- Unauthorized: (<zope.app.publisher.browser.viewmeta.SnarfCommit ...>, ...
+ Unauthorized: ...
TODO:
Copied: zope.app.fssync/trunk/src/zope/app/fssync/ssh.py (from rev 97911, zope.app.fssync/branches/jim-hack/src/zope/app/fssync/ssh.py)
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/ssh.py (rev 0)
+++ zope.app.fssync/trunk/src/zope/app/fssync/ssh.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -0,0 +1,138 @@
+##############################################################################
+#
+# Copyright (c) 2009 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+import getpass
+import httplib
+import os.path
+import socket
+
+import paramiko
+
+
+class FileSocket(object):
+ """
+ Adapts a file to the socket interface for use with http.HTTPResponse
+ """
+ def __init__(self, file):
+ self.file = file
+
+ def makefile(self, *args):
+ return self.file
+
+
+class SSHConnection(object):
+ """
+ SSH connection that implements parts of the httplib.HTTPConnection
+ interface
+ """
+ def __init__(self, host_port, user_passwd=None):
+ self.headers = {}
+ self.host, self.port = host_port.split(':')
+ self.port = int(self.port)
+
+ # if username is specified in URL then use it, otherwise
+ # default to local userid
+ if user_passwd:
+ self.remote_user_name = user_passwd.split(':')[0]
+ else:
+ self.remote_user_name = getpass.getuser()
+
+ def putrequest(self, method, path):
+ # open connection to server
+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ sock.connect((self.host, self.port))
+ self.transport = paramiko.Transport(sock)
+ self.transport.start_client()
+
+ # try to get public key from ssh agent
+ agent = paramiko.Agent()
+ for key in agent.get_keys():
+ try:
+ self.transport.auth_publickey(self.remote_user_name, key)
+ break
+ except paramiko.SSHException:
+ pass
+
+ # try to get public key from fs
+ if not self.transport.is_authenticated():
+ path = os.path.expanduser('~/.ssh/id_rsa')
+ try:
+ key = paramiko.RSAKey.from_private_key_file(path)
+ except paramiko.PasswordRequiredException:
+ password = getpass.getpass('RSA key password: ')
+ key = paramiko.RSAKey.from_private_key_file(path, password)
+ try:
+ self.transport.auth_publickey(self.remote_user_name, key)
+ except paramiko.SSHException:
+ pass
+
+ if not self.transport.is_authenticated():
+ path = os.path.expanduser('~/.ssh/id_dsa')
+ try:
+ key = paramiko.DSSKey.from_private_key_file(path)
+ except paramiko.PasswordRequiredException:
+ password = getpass.getpass('DSS key password: ')
+ key = paramiko.DSSKey.from_private_key_file(path, password)
+ try:
+ self.transport.auth_publickey(self.remote_user_name, key)
+ except paramiko.SSHException:
+ raise Exception('No valid public key found')
+
+ # try to get host key
+ hostkeytype = None
+ hostkey = None
+ try:
+ host_keys = paramiko.util.load_host_keys(
+ os.path.expanduser('~/.ssh/known_hosts'))
+ except IOError:
+ try:
+ # try ~/ssh/ too, because windows can't have a folder
+ # named ~/.ssh/
+ host_keys = paramiko.util.load_host_keys(
+ os.path.expanduser('~/ssh/known_hosts'))
+ except IOError:
+ host_keys = {}
+
+ if host_keys.has_key(self.host):
+ hostkeytype = host_keys[self.host].keys()[0]
+ hostkey = host_keys[self.host][hostkeytype]
+
+ # verify host key
+ if hostkey:
+ server_key = self.transport.get_server_key()
+ if server_key != hostkey:
+ raise Exception(
+ "Remote host key doesn't match value in known_hosts")
+
+ # start zsync subsystem on server
+ self.channel = self.transport.open_session()
+ self.channel.invoke_subsystem('zsync')
+ self.channelr = self.channel.makefile('rb')
+ self.channelw = self.channel.makefile('wb')
+
+ # start sending request
+ self.channelw.write('%s %s\r\n' % (method, path))
+
+ def putheader(self, name, value):
+ self.channelw.write('%s: %s\r\n' % (name, value))
+
+ def endheaders(self):
+ self.channelw.write('\r\n')
+
+ def send(self, data):
+ self.channelw.write(data)
+
+ def getresponse(self):
+ response = httplib.HTTPResponse(FileSocket(self.channelr))
+ response.begin()
+ return response
Modified: zope.app.fssync/trunk/src/zope/app/fssync/syncer.py
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/syncer.py 2009-03-11 19:11:52 UTC (rev 97911)
+++ zope.app.fssync/trunk/src/zope/app/fssync/syncer.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -52,7 +52,7 @@
factory = component.queryUtility(interfaces.ISynchronizerFactory)
if factory is None:
if raise_error:
- raise synchronizer.MissingSerializer(dn)
+ raise synchronizer.MissingSynchronizer(dn)
return None
checker = getattr(factory, '__Security_checker__', None)
@@ -79,4 +79,3 @@
checkout.perform(obj, name)
return temp
-
\ No newline at end of file
Copied: zope.app.fssync/trunk/src/zope/app/fssync/tests/ssh.txt (from rev 97911, zope.app.fssync/branches/jim-hack/src/zope/app/fssync/tests/ssh.txt)
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/tests/ssh.txt (rev 0)
+++ zope.app.fssync/trunk/src/zope/app/fssync/tests/ssh.txt 2009-03-11 20:04:22 UTC (rev 97912)
@@ -0,0 +1,119 @@
+SSH Networking
+==============
+
+The ssh module provides ssh networking classes. The main class is
+SSHConnection, which follows the httplib.HTTPConnection interface.
+
+This class don't provide complete compatibility with httplib; it only
+implements the parts that zope.app.fssync expects.
+
+
+Response
+--------
+
+In order to handle reponses the ssh connection uses an adapter that
+makes a file look like a socket. This adapter is then used to create
+an HTTPResponse.
+
+ >>> from zope.app.fssync.ssh import FileSocket
+ >>> from httplib import HTTPResponse
+
+ >>> data = """HTTP/1.0 302 Found\r
+ ... Location: http://www.google.ca/\r
+ ... Cache-Control: private\r
+ ... Content-Type: text/html; charset=UTF-8\r
+ ... Set-Cookie: PREF=ID=6bcca0d3fbdb7918:TM=1229706725:LM=1229706725:S=Io4QKPOm7laH7W7q; expires=Sun, 19-Dec-2010 17:12:05 GMT; path=/; domain=.google.com\r
+ ... Date: Fri, 19 Dec 2008 17:12:05 GMT\r
+ ... Server: gws\r
+ ... Content-Length: 218\r
+ ... Connection: Close\r
+ ... \r
+ ... <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
+ ... <TITLE>302 Moved</TITLE></HEAD><BODY>
+ ... <H1>302 Moved</H1>
+ ... The document has moved
+ ... <A HREF="http://www.google.ca/">here</A>.
+ ... </BODY></HTML>"""
+
+ >>> from cStringIO import StringIO
+ >>> f = StringIO()
+ >>> f.write(data)
+ >>> f.seek(0)
+ >>> s = FileSocket(f)
+ >>> r = HTTPResponse(s)
+ >>> r.begin()
+
+Now let's make sure that the response looks right.
+
+ >>> r.msg['Content-type']
+ 'text/html; charset=UTF-8'
+
+ >>> r.reason
+ 'Found'
+
+ >>> r.fp.read()
+ '<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n<TITLE>302 Moved</TITLE></HEAD><BODY>\n<H1>302 Moved</H1>\nThe document has moved\n<A HREF="http://www.google.ca/">here</A>.\n</BODY></HTML>'
+
+Excellent.
+
+Connection
+----------
+
+The connection class should be tested here. This will be hard since it
+requires lots of stuff like having ssh agent running, host keys, and
+having an ssh server running. For now let's just do what we can.
+
+ >>> from zope.app.fssync.ssh import SSHConnection
+ >>> c = SSHConnection('localhost:12345')
+ >>> c.putrequest('GET', '/')
+ Traceback (most recent call last):
+ ...
+ error: (111, 'Connection refused')
+
+We get an error here because we're not running an ssh server on port
+12345.
+
+Even though we don't have a real encrypted network connection, let's
+see what the connection object would send over the ssh channel.
+ >>> class StubFile(object):
+ ... def write(self, data):
+ ... print repr(data)
+ >>> class StubChannel(object):
+ ... def close(self): pass
+ >>> c.channel = StubChannel()
+ >>> c.channelw = StubFile()
+ >>> c.putheader('Header Name', 'Header value')
+ 'Header Name: Header value\r\n'
+
+ >>> c.endheaders()
+ '\r\n'
+
+ >>> c.send('this is some data')
+ 'this is some data'
+
+The connection understands data sent back from the ssh server as a
+HTTP response. We can demo this by using a fake server response.
+
+ >>> from cStringIO import StringIO
+ >>> f = StringIO("""HTTP/1.0 200 OK\r\nHeader: value\r\n\r\nBody""")
+ >>> f.seek(0)
+ >>> c.channelr = f
+ >>> class StubTransport(object):
+ ... def close(self): pass
+ >>> c.transport = StubTransport()
+
+Now we can test the response.
+
+ >>> r = c.getresponse()
+ >>> r
+ <httplib.HTTPResponse instance at 0x...>
+
+ >>> r.getheaders()
+ [('header', 'value')]
+
+ >>> r.status
+ 200
+
+ >>> r.read()
+ 'Body'
+
Copied: zope.app.fssync/trunk/src/zope/app/fssync/tests/test_ssh.py (from rev 97911, zope.app.fssync/branches/jim-hack/src/zope/app/fssync/tests/test_ssh.py)
===================================================================
--- zope.app.fssync/trunk/src/zope/app/fssync/tests/test_ssh.py (rev 0)
+++ zope.app.fssync/trunk/src/zope/app/fssync/tests/test_ssh.py 2009-03-11 20:04:22 UTC (rev 97912)
@@ -0,0 +1,21 @@
+##############################################################################
+#
+# Copyright (c) 2009 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+import doctest
+import unittest
+
+def test_suite():
+ return unittest.TestSuite([
+ doctest.DocFileSuite('ssh.txt',
+ optionflags=(doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE))
+ ])
More information about the Checkins
mailing list