[Zope-dev] Git conversion with svn-all-fast-export

[Marius Gedminas]

You need:

- svn-all-fast-export (from the Debian/Ubuntu package of the same name;
  upstream homepage is http://gitorious.org/svn2git, not related to a
  Ruby tool of the same name)

- a copy of the Subversion repository

    Good thing I have one set up, using svnsync:

      # http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.replication
      svnadmin create /stuff/zope-mirror
      svnadmin setuuid /stuff/zope-mirror 62d5b8a3-27da-0310-9561-8e5933582275
      svnsync init file:///stuff/zope-mirror svn://svn.zope.org/repos/main/
      svnsync sync file:///stuff/zope-mirror
      # repeat last command periodically

    It needs about 3.3 gigs of disk space.

- a copy of authors.txt that maps svn usernames to real names and emails
  (ask Tres or Jim)

- an empty repository on Github (ask Tres or Jim to create one at
  https://github.com/zopefoundation)

    Speaking of this, could somebody create a zope.dottedname repo?  Thanks.


The conversion process goes like this:

- write a rules.txt like this one I used for zope.dottedname:

    create repository zope.dottedname
    end repository

    # feel free to create multiple repositories in one go

    # order of matches matters in this file
    # trailing slashes in match rules are very important

    match /
      # skip revisions 129030, 129031 and 129032, since they're empty
      # in my svn mirror (no files, no committer, no date) for some reason
      # (bug in svnsync?), and they make svn-all-fast-export segfault
      min revision 129030
      max revision 129032
    end match

    match /(zope\.dottedname)/trunk/
      repository \1
      branch master
    end match

    match /(zope\.dottedname)/branches/([^/]+)/
      repository \1
      branch \2
    end match

    match /(zope\.dottedname)/tags/([^/]+)/
      repository \1
      branch refs/tags/\2
    end match

    match /
      # ignore all other projects
    end match

- run svn-all-fast-export --identity-map=authors.txt --rules=rules.txt --stats \
      /path/to/your/zope-svn-mirror

- wait a bit

    The first time I ran it it took ~18 wall clock minutes (~4 CPU
    minutes) and ended in a segfault.  The second run took 12 wall clock
    minutes (hot disk cache, I suppose) and also ended in a segfault.
    Then I discovered that if I don't remove the git repository,
    svn-all-fast-export will resume the process (a few thousand
    revisions before it crashed, or maybe that just happened to be the
    last successfully converted commit before the crash), which is
    considerably faster than starting from scratch.

- inspect ./zope.dottedname for sanity

    I recommend tig as a very nice console-mode interactive git history viewer.

    For an example of things to inspect, e.g., there was a deleted 3.4.1 tag
    from http://zope3.pov.lt/trac/changeset/80495, which shouldn't have been
    deleted, according to http://zope3.pov.lt/trac/changeset/80499, so I've
    re-created the tag from refs/backups/r80495/tags/3.4.1 that was left by
    the conversion tool.

- upload to github


Advantages of svn-all-fast-export

  - it's fast (<5 CPU minutes for 129128 svn revisions; I wish I had an SSD on
    the server that hosts my svn mirror)

  - it can simultaneously convert multiple packages (add more 'create
    repository/end repository' statements to rules.txt, and extend the match
    regexps to catch the packages you're interested in)

  - it's very flexible and can handle gnarly repository history, if you write
    rules for it

  - it was written for and used by the KDE project to convert and
    explode their humongous svn repository into a multitude of git
    projects, so it's been stress-tested rather well

Disadvantages of svn-all-fast-export

  - it requires a local copy of the entire subversion repository

  - it doesn't produce error messages if something's wrong, instead it
    segfaults


Incidentally, I'd like to hear reports about conversion speed from
people (e.g. Jim ;-) who have used Jim's conversion script.  Is 12
minutes fast or slow?  zope.dottedname is not a very large package; it
had merely 34 revisions in trunk's history.


I would also like to offer my services as a git conversion specialist,
since I have access to an svn mirror.  Ask me for a package by name, and
I'll send you a git repository in a few days.  It'll be up to you to
inspect the fidelity of the conversion, though; I don't have the spare
cycles for that.


Regards,
Marius Gedminas
-- 
http://pov.lt/ -- Zope 3/BlueBream consulting and development
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zope-dev/attachments/20130204/195768ea/attachment.sig>