[ZODB-Dev] RelStorage branch 1.4.0-fastimport

[Shane Hathaway]

On 01/11/2011 07:06 AM, David Blewett wrote:
> On Thu, Jan 6, 2011 at 11:51 PM, Shane Hathaway<shane at hathawaymix.org>  wrote:
>>> I am no zodb or relstorage expert, but the changes for the
>>> --single-transaction code look good to me.
>>
>> I looked at that branch before, but I felt like the changes were
>> complicated enough to require a comparison with simpler solutions first.
>>   In particular, Postgres has an option to disable fsync.  Set it in
>> postgresql.conf.  Disabling fsync is not normally recommended, but for a
>> large import it's obviously a good idea.  Would you compare the speed of
>> --single-transaction with disabled fsync on vanilla RelStorage?
>
> Sorry, it's been awhile since I started work on that branch. Disabling
> fsync is an option for the load, and it should help. However, using a
> single transaction is much faster in PostgreSQL. Off the top of my
> head, I can't recall if the branch also uses COPY mode. I recall
> encountering some problems trying to get COPY to work correctly.
> Single-transaction mode + COPY, however, is the fastest way to get
> data into PostgreSQL.

Right, but I want zodbconvert to remain a very generic tool.  I think of 
zodbconvert as a reference implementation.  A tool that uses PostgreSQL 
COPY is a good idea but it ought to be a different tool.

IMHO, it's actually pretty easy to write optimized ZODB conversion 
tools.  I would suggest writing conversion tools as independent 
packages, meaning they probably should not import RelStorage, but 
instead talk to the database directly.

Shane