[Zope3-dev] RE: [Zope3-checkins] SVN: Zope3/trunk/src/Tell subversion to expandthe Id keyword:

[Tim Peters]

[Jim Fulton]
> Right. This is a good thing.

Yes, but it's a mixed thing, not purely good.  It has good points and bad
points,

> The real ZODB, ZConfig and zdaemon projects were unaffected by Philipp's
> changes. He didn't need to worry about implicitly changing a separate
> project.

That's no benefit in this case.  We *want* to do Id expansion in all copies
of ZODB, and it's a simple mechanical change.  It would have done no harm at
all, but would have saved additional steps, if my changing ZODB to do Id
expansion had affected all projects using ZODB -- or if Philipp's attempt in
Zope3 would have affected all projects using ZODB.  Now I'm left unsure what
state ZODB is in anywhere (except in the ZODB repository, where I'm the only
one who has mucked with it).

> Worse implies that there is a problem to begin with. I don't agree.

The ZODB in ZODB is no longer the same as the ZODB in Zope3 HEAD is no
longer the same as the ZODB in Zope HEAD.  We've been here before, and over
time it gets worse.  I've seen more ZODB bugs in my time here due to
screwing up synchs across a seemingly endless number of active ZODB branches
than due to any other single cause that I can recall.  That's a serious
problem to me, and, as I said before, things got much better in this respect
when we cut the number of active ZODB branches in CVS in half (by
eliminating the distinctions between ZODB HEAD and Zope HEAD, ZODB 3.2 and
Zope 2.7 branch, ZODB 3.1 and Zope 2.6 branch).

>> and in the ZODB case a merge with the real ZODB code is going to yield
>> conflicts on every .py, .c and .h file.

> Why would it?

I thought it would because Philipp changed the same lines in the same files
that I changed.  But you're right that svn isn't in fact calling these
conflicts.  I don't understand what svn *is* in fact doing, but calling them
conflicts it ain't.

> I tried:
> 
>    $ svn merge \
>          svn+ssh://svn.zope.org/repos/main/ZODB/trunk/src/ZODB \
>
svn+ssh://svn.zope.org/repos/main/ZODB/tags/cvs-to-svn-conversion/src/ZODB \
>          .

As we discussed, I think this is backwards:  it's telling svn to take all
the changes made in ZODB since the cvs-to-svn conversion, and *remove* them
from Zope3 (assuming "." there means your Zope3 src/ZODB directory).  That's
one way merges routinely screw things up, BTW -- merges are delicate and so
error-prone.

I don't understand svn's output, and it's pretty much the same even if the
first two arguments to the merge above are reversed.  For one thing, it
*only* lists .py files, but Id expansion was also added to .c and .h files:

>     U zodb4/z4base.py
>     U zodb4/z4errors.py
>     U zodb4/tests/test_z4utils.py
>     U zodb4/tests/__init__.py
>     U zodb4/z4utils.py
>     U zodb4/__init__.py
>    UU zodb4/z4interfaces.py
>     U zodb4/main.py
>     U zodb4/conversion.py
>     U zodb4/z4format.py
>     U zodb4/z4iterator.py
>     U fsrecover.py
>    UU Connection.py
>    UU BaseStorage.py
>     U __init__.py
>    UU winlock.c
>     U fsIndex.py
>    UU ActivityMonitor.py
>    UU config.py
>     U UndoLogCompatible.py
>     U DemoStorage.py
>     U tests/testmvcc.py
>     U tests/testZODB.py
>     U tests/testSubTransaction.py
>     U tests/__init__.py
>     U tests/Synchronization.py
>     U tests/ReadOnlyStorage.py
>    UU tests/sampledm.py
>     U tests/test_storage.py
>     U tests/Corruption.py
>     U tests/testFileStorage.py
>     U tests/testConfig.py
>     U tests/BasicStorage.py
>    UU tests/util.py
>     U tests/TransactionalUndoStorage.py
>     U tests/LocalStorage.py
>     U tests/MinPO.py
>     U tests/RecoveryStorage.py
>     U tests/testSerialize.py
>     U tests/warnhook.py
>     U tests/dangle.py
>     U tests/testPersistentList.py
>     U tests/testConnection.py
>     U tests/StorageTestBase.py
>     U tests/speed.py
>     U tests/testUtils.py
>     U tests/testfsIndex.py
>     U tests/test_cache.py
>     U tests/testCache.py
>     U tests/testTimeStamp.py
>    UU tests/test_datamanageradapter.py
>    UU tests/testActivityMonitor.py
>     U tests/MTStorage.py
>     U tests/testRecover.py
>     U tests/testDemoStorage.py
>     U tests/VersionStorage.py
>     U tests/testPersistentMapping.py
>     U tests/IteratorStorage.py
>    UU tests/testBroken.py
>     U tests/testMappingStorage.py
>     U tests/RevisionStorage.py
>     U tests/TransactionalUndoVersionStorage.py
>     U tests/testDB.py
>     U tests/PersistentStorage.py
>     U tests/ConflictResolution.py
>     U tests/HistoryStorage.py
>     U tests/PackableStorage.py
>     U TmpStore.py
>     U transact.py
>     U MappingStorage.py
>    UU interfaces.py
>    UU DB.py
>     U serialize.py
>     U dbmStorage.py
>     U fstools.py
>     U loglevels.py
>     U ZApplication.py
>     U utils.py
>    UU Mount.py
>     U ExportImport.py
>     U conversionhack.py
>     U FileStorage/format.py
>     U FileStorage/fsdump.py
>     U FileStorage/__init__.py
>     U FileStorage/FileStorage.py
>     U FileStorage/fspack.py
>    UU broken.py
>    UU POSException.py
>     U ConflictResolution.py
>     U lock_file.py

No .c or .h files there.  An example of a .c file not listed:

C:\Code>svn proplist \
svn://svn.zope.org/repos/main/ZODB/trunk/src/BTrees/BTreeItemsTemplate.c
Properties on
'svn://svn.zope.org/repos/main/ZODB/trunk/src/BTrees/BTreeItemsTemplate.c':
  cvs2svn:cvs-rev
  svn:keywords
  svn:eol-style

So keywords is set in "the real" ZODB.

C:\Code>svn proplist \
 svn://svn.zope.org/repos/main/Zope3/trunk/src/BTrees/BTreeItemsTemplate.c
Properties on
'svn://svn.zope.org/repos/main/Zope3/trunk/src/BTrees/BTreeItemsTemplate.c':
  cvs2svn:cvs-rev
  svn:keywords
  svn:eol-style

And keywords is set in Zope3's version of the same file.

C:\Code>svn proplist
svn://svn.zope.org/repos/main/ZODB/tags/cvs-to-svn-conversion/src/BTrees/BTr
eeItemsTemplate.c
Properties on
'svn://svn.zope.org/repos/main/ZODB/tags/cvs-to-svn-conversion/src/BTrees/BT
reeItemsTemplate.c':
  cvs2svn:cvs-rev
  svn:eol-style

But keywords is not set on the original ZODB conversion to svn.

So applying the merge backwards tells svn to undo Philipp's Id expansion in
Zope3's version of the file, yet svn doesn't produce any output about that.

If I do the merge "forwards", then I see svn *does* mention this file:

    UU BTrees\BTreeItemsTemplate.c

but it doesn't actually change anything in it (the working copy after the
merge is identical to the one in the repositories (both ZODB and Zope3),
both text and set of keywords).

The forward direction also produces a bunch of nuisance (I think -- unsure)
output, like (this is a small part of it):

Skipped missing target: 'scripts\fsrefs.py'
Skipped missing target: 'scripts\migrate.py'
Skipped missing target: 'scripts\parsezeolog.py'
Skipped missing target: 'scripts\space.py'
Skipped missing target: 'scripts\zeopack.py'
Skipped missing target: 'scripts\zodbload.py'
Skipped missing target: 'scripts\fsdump.py'
Skipped missing target: 'scripts\zeoqueue.py'
Skipped missing target: 'scripts\netspace.py'
Skipped missing target: 'scripts\zeoup.py'

and records some conflicts:

CU BTrees\tests\testBTreesUnicode.py
CU ZODB\ActivityMonitor.py
CU ZODB\Mount.py

>    $ svn status
>    MM     zodb4/z4interfaces.py
>    MM     Connection.py
>    MM     BaseStorage.py
>    MM     winlock.c
>    MM     ActivityMonitor.py
>    MM     config.py
>    MM     tests/sampledm.py
>    MM     tests/util.py
>    MM     tests/test_datamanageradapter.py
>    MM     tests/testActivityMonitor.py
>    MM     tests/testBroken.py
>    MM     interfaces.py
>    MM     DB.py
>    MM     Mount.py
>    MM     broken.py
>    MM     POSException.py
>
> Worked great.

That's because it was backwards <wink>.  After a forward merge, it's
messier:

C:\Code\Zope3\src>svn st
M      persistent\mapping.py
 M     persistent\ring.h
M      persistent\list.py
 M     persistent\tests\test_list.py
 M     persistent\tests\persistenttestbase.py
 M     persistent\tests\test_persistent.py
 M     persistent\tests\__init__.py
 M     persistent\tests\testPersistent.py
 M     persistent\cPersistence.h
 M     BTrees\IOBTree.py
 M     BTrees\OIBTree.py
 M     BTrees\OOBTree.py
 M     BTrees\__init__.py
?      BTrees\tests\testBTreesUnicode.py.working
?      BTrees\tests\testBTreesUnicode.py.merge-left.r24795
?      BTrees\tests\testBTreesUnicode.py.merge-right.r24795
C      BTrees\tests\testBTreesUnicode.py
 M     BTrees\tests\test_check.py
 M     BTrees\tests\testSetOps.py
 M     BTrees\tests\__init__.py
 M     BTrees\tests\test_btreesubclass.py
 M     BTrees\tests\testConflict.py
 M     BTrees\tests\testBTrees.py
 M     BTrees\tests\test_compare.py
 M     BTrees\convert.py
 M     BTrees\check.py
 M     BTrees\Length.py
 M     BTrees\IIBTree.py
 M     BTrees\Interfaces.py
 M     transaction\_manager.py
 M     transaction\tests\__init__.py
 M     transaction\tests\test_register_compat.py
 M     transaction\__init__.py
 M     transaction\_transaction.py
 M     ZEO\simul.py
 M     ZEO\CommitLog.py
 M     ZEO\StorageServer.py
 M     ZEO\stats.py
 M     ZEO\__init__.py
 M     ZEO\zeoctl.py
 M     ZEO\ClientStorage.py
 M     ZEO\cache.py
 M     ZEO\ServerStub.py
 M     ZEO\auth\hmac.py
 M     ZEO\auth\base.py
 M     ZEO\auth\__init__.py
 M     ZEO\auth\auth_digest.py
 M     ZEO\tests\forker.py
 M     ZEO\tests\auth_plaintext.py
 M     ZEO\tests\multi.py
 M     ZEO\tests\ThreadTests.py
 M     ZEO\tests\testZEO.py
 M     ZEO\tests\__init__.py
 M     ZEO\tests\testConnection.py
 M     ZEO\tests\speed.py
 M     ZEO\tests\testAuth.py
 M     ZEO\tests\test_cache.py
 M     ZEO\tests\Cache.py
 M     ZEO\tests\InvalidationTests.py
 M     ZEO\tests\deadlock.py
 M     ZEO\tests\TestThread.py
 M     ZEO\tests\testZEOOptions.py
 M     ZEO\tests\testTransactionBuffer.py
 M     ZEO\tests\ConnectionTests.py
 M     ZEO\tests\stress.py
 M     ZEO\tests\CommitLockTests.py
 M     ZEO\tests\zeoserver.py
 M     ZEO\ClientStub.py
 M     ZEO\util.py
 M     ZEO\DebugServer.py
 M     ZEO\TransactionBuffer.py
 M     ZEO\zeopasswd.py
 M     ZEO\Exceptions.py
 M     ZEO\runzeo.py
 M     ZEO\mkzeoinst.py
 M     ZEO\zrpc\error.py
 M     ZEO\zrpc\client.py
 M     ZEO\zrpc\__init__.py
 M     ZEO\zrpc\connection.py
 M     ZEO\zrpc\log.py
 M     ZEO\zrpc\smac.py
 M     ZEO\zrpc\trigger.py
 M     ZEO\zrpc\server.py
 M     ZEO\zrpc\marshal.py
 M     ZEO\zrpc\_hmac.py
?      zope\app\mail\tests\mailbox
?      ZODB\ActivityMonitor.py.merge-right.r24795
?      ZODB\Mount.py.merge-left.r24795
?      ZODB\Mount.py.working
?      ZODB\Mount.py.merge-right.r24795
?      ZODB\ActivityMonitor.py.merge-left.r24795
?      ZODB\ActivityMonitor.py.working
 M     ZODB\zodb4\z4base.py
 M     ZODB\zodb4\z4errors.py
 M     ZODB\zodb4\tests\test_z4utils.py
 M     ZODB\zodb4\tests\__init__.py
 M     ZODB\zodb4\z4utils.py
 M     ZODB\zodb4\__init__.py
 M     ZODB\zodb4\main.py
 M     ZODB\zodb4\conversion.py
 M     ZODB\zodb4\z4format.py
 M     ZODB\zodb4\z4iterator.py
 M     ZODB\fsrecover.py
 M     ZODB\__init__.py
 M     ZODB\fsIndex.py
C      ZODB\ActivityMonitor.py
 M     ZODB\UndoLogCompatible.py
MM     ZODB\DemoStorage.py
 M     ZODB\tests\testmvcc.py
 M     ZODB\tests\testZODB.py
 M     ZODB\tests\testSubTransaction.py
 M     ZODB\tests\__init__.py
 M     ZODB\tests\Synchronization.py
 M     ZODB\tests\ReadOnlyStorage.py
 M     ZODB\tests\test_storage.py
 M     ZODB\tests\Corruption.py
 M     ZODB\tests\testFileStorage.py
 M     ZODB\tests\testConfig.py
 M     ZODB\tests\BasicStorage.py
 M     ZODB\tests\TransactionalUndoStorage.py
 M     ZODB\tests\LocalStorage.py
 M     ZODB\tests\MinPO.py
 M     ZODB\tests\RecoveryStorage.py
 M     ZODB\tests\testSerialize.py
 M     ZODB\tests\warnhook.py
 M     ZODB\tests\dangle.py
 M     ZODB\tests\testPersistentList.py
 M     ZODB\tests\StorageTestBase.py
 M     ZODB\tests\testConnection.py
 M     ZODB\tests\testfsIndex.py
 M     ZODB\tests\testUtils.py
 M     ZODB\tests\speed.py
 M     ZODB\tests\test_cache.py
 M     ZODB\tests\testTimeStamp.py
 M     ZODB\tests\testCache.py
 M     ZODB\tests\MTStorage.py
 M     ZODB\tests\testRecover.py
 M     ZODB\tests\testDemoStorage.py
 M     ZODB\tests\VersionStorage.py
 M     ZODB\tests\testPersistentMapping.py
 M     ZODB\tests\IteratorStorage.py
 M     ZODB\tests\testMappingStorage.py
 M     ZODB\tests\RevisionStorage.py
 M     ZODB\tests\TransactionalUndoVersionStorage.py
 M     ZODB\tests\testDB.py
 M     ZODB\tests\PersistentStorage.py
 M     ZODB\tests\ConflictResolution.py
 M     ZODB\tests\HistoryStorage.py
 M     ZODB\tests\PackableStorage.py
 M     ZODB\TmpStore.py
 M     ZODB\transact.py
MM     ZODB\MappingStorage.py
MM     ZODB\dbmStorage.py
 M     ZODB\serialize.py
 M     ZODB\fstools.py
 M     ZODB\loglevels.py
MM     ZODB\ZApplication.py
 M     ZODB\utils.py
C      ZODB\Mount.py
 M     ZODB\ExportImport.py
 M     ZODB\conversionhack.py
 M     ZODB\FileStorage\format.py
 M     ZODB\FileStorage\fsdump.py
 M     ZODB\FileStorage\__init__.py
 M     ZODB\FileStorage\FileStorage.py
 M     ZODB\FileStorage\fspack.py
 M     ZODB\ConflictResolution.py
 M     ZODB\lock_file.py
MM     ThreadedAsync\LoopCallback.py

Those are real conflicts, BTW.  I got rid of __version__ globals in "the
real" ZODB, but Philipp changed them from Rev to Id expansions.  So he
wasted some of his time doing that, and I'll waste more of mine (one way or
another ... I already burned time finding out why "C" was produced) undoing
that.

> Second, if there are no substantive changes in Zope 3's copy of ZODB,
> which there should not be, then we will probably choose to recopy
> rather than merge.
>
> Note that it's easy to find out that there were no substantive changes:

Sorry, I don't trust reading log messages to determine the extent of code
changes.
 
> I don't see a problem here. In fact, it prevented a headache, because
> Philipp's changes didn't accidentally affect the ZODB project.

If my setting of ZODB Id expansion had taken effect in Zope3, then Philipp
wouldn't have needed to change the Zope3 versions, and we wouldn't need to
talk about how to finish the job now.  And, AFAIK, nobody yet is "in charge"
of the Zope svn repository in the way you are for Zope3.  It's not all
roses!  It's not all thistles either.

> I don't agree.  We have two sane ways: merging and recopying.

>From my "forward" experiment above, I'm doubting that merging is sane,
unless ZODB is left entirely alone in the projects that "borrow" it.  There
indeed haven't been *substantive* changes yet, but there are already
mysteries and conflicts.

> ...
> Now, substantive changes are potentially more problematic.  Someday,
> we'll probably need to make some ZODB fix to some Zope release branch.
> Such a fix will be very explicit.  It should also be easy to either
> merge such a fix back into ZODB or to apply some similar fix to ZODB and
> resync.

Maybe.  History says "similar fix" has an extraordinarily high chance of
failing in an unanticipated way.  Distinct copies of code always get out of
whack, and pasting them together again is always a potential source of new
problems.  The "unified" experience on the Zope 2.6 and 2.7 branches (vs
their respective versions of ZODB) has been very good so far.  The svn
experience has also been very good so far, but it's a much shorter "so far"
than the 2.6/2.7 experience, and has seen far fewer substantive ZODB
changes.

> I actually think that the more common way to sync up will be to recopy.
> The only thing that would prevent that would be if there are zope-specific
> changes to ZODB that were never applied to the original ZODB project. We
> wouldn't do that, because it wouldn't be sane.

If "we" means you and me, I believe that.  I'm not sure everyone understands
how ZODB and ZConfig get stitched into other projects; those who don't won't
know that there's a reason to take care in those directories.

> There's nothing to untangle.

Maybe not for you <wink>.