[ZODB-Dev] RelStorage branch 1.4.0-fastimport

[Shane Hathaway]

On 01/06/2011 03:35 PM, Maurits van Rees wrote:
> davidb started a branch of RelStorage in July 2010 and I wonder if that
> can be merged.  It does two interesting things in the zodbconvert code:
>
> - It adds a --single-transaction command line option.  From the readme:
>     "Import into the destination in a single transaction, instead of one
>     transaction per transaction in the source. This option can
>     significantly speed up conversion times on PostgreSQL."
>
> - I adds more logging so you have an indication that yes, something is
> in fact happening.  This is good for your heart when you want to migrate
> a 16 GB Data.fs. :-)  Otherwise you just see 'Storages opened
> successfully'  and then nothing at all for several hours.
>
> I am no zodb or relstorage expert, but the changes for the
> --single-transaction code look good to me.

I looked at that branch before, but I felt like the changes were 
complicated enough to require a comparison with simpler solutions first. 
  In particular, Postgres has an option to disable fsync.  Set it in 
postgresql.conf.  Disabling fsync is not normally recommended, but for a 
large import it's obviously a good idea.  Would you compare the speed of 
--single-transaction with disabled fsync on vanilla RelStorage?

I assume the logging additions are less invasive, and if that assumption 
is correct, there's no problem with merging those.

> I added some more logging on that branch, mostly because the conversion
> appeared to be hanging at some unknown point.  This was also with the
> official 1.4.1 release, which was the reason I started experimenting
> with the fastimport branch to see if that would help.  It did not.  At
> least in both test runs mentioned above, the actual time it took was
> about twenty minutes longer, possibly because the conversion temporarily
> lost the connection with the postgres server.  With the logging I could
> at least see that it was throwing the old transaction table away; I have
> seen the same with other tables.  Definitely no one else is accessing
> this database at the same time.  So if someone has an idea what could be
> going on here, that is welcome.

Are you sure you were not accidentally running multiple imports in 
parallel?  RelStorage does not throw away tables unless you're rather 
explicit about it.

> Anyway, the --single-transaction seems to work and I would say the
> logging is helpful.  So: is there any chance this can be merged to
> trunk?

Not yet.  We really need someone to do the fsync test, and if that 
doesn't do the trick, there has to be a way to accomplish the same thing 
in a clearer way.

>  I am also interested in the blob support that has been added
> there. :-)

The blob changes are also in 1.5.0a1.  That release seems to be a lot 
more stable than I expected and may soon become 1.5.0 final if no one 
finds bugs in the new blob option.

Shane