[ZODB-Dev] RelStorage branch 1.4.0-fastimport

[Maurits van Rees]

Hi all,

davidb started a branch of RelStorage in July 2010 and I wonder if that 
can be merged.  It does two interesting things in the zodbconvert code:

- It adds a --single-transaction command line option.  From the readme:
   "Import into the destination in a single transaction, instead of one
   transaction per transaction in the source. This option can
   significantly speed up conversion times on PostgreSQL."

- I adds more logging so you have an indication that yes, something is 
in fact happening.  This is good for your heart when you want to migrate 
a 16 GB Data.fs. :-)  Otherwise you just see 'Storages opened 
successfully'  and then nothing at all for several hours.

I am no zodb or relstorage expert, but the changes for the 
--single-transaction code look good to me.

In practice it seems to work too.  I converted a CatalogData.fs of 150 
MB to postgres.  With --single-transaction the final result printed was:

All 2635 transactions copied successfully in  4.3 minutes at 0.550mB/s.

Without that option the result was:

All 2635 transactions copied successfully in  6.2 minutes at 0.379mB/s.

This is about a 30% decrease in time using the --single-transaction option.

In both cases the target postgres database was already filled so I used 
the --clear option as well.  The conversion was done on the server that 
had the CatalogData.fs.  The postgres database was hosted on a different 
server in the same network.

I added some more logging on that branch, mostly because the conversion 
appeared to be hanging at some unknown point.  This was also with the 
official 1.4.1 release, which was the reason I started experimenting 
with the fastimport branch to see if that would help.  It did not.  At 
least in both test runs mentioned above, the actual time it took was 
about twenty minutes longer, possibly because the conversion temporarily 
lost the connection with the postgres server.  With the logging I could 
at least see that it was throwing the old transaction table away; I have 
seen the same with other tables.  Definitely no one else is accessing 
this database at the same time.  So if someone has an idea what could be 
going on here, that is welcome.


Anyway, the --single-transaction seems to work and I would say the 
logging is helpful.  So: is there any chance this can be merged to 
trunk?  I am also interested in the blob support that has been added 
there. :-)

Thanks,


-- 
Maurits van Rees
Programmer, Zest Software