[ZODB-Dev] minimizing the need for database packing

[Jim Fulton]

On Dec 29, 2007, at 3:22 PM, Dieter Maurer wrote:

> Jim Fulton wrote at 2007-12-28 10:20 -0500:
>> ...
>> There Berkely Database Storage supported automatic incremental  
>> packing
>> without garbage collection.  If someone were to revitalize that  
>> effort
>> and if one was willing to do without cyclic garbage collection, then
>> that storage would remove the need for the sort of disruptive pack we
>> have with FileStorage now.
>
> Why do you consider "pack" disruptive?

Disruptive isn't exactly the word I was looking for, although the  
current pack approach is disruptive.
It causes a concentrated I/O and CPU load.  For our databases,  
database performance drops significantly, especially during the later  
stages of packing.



>> Note that I'm working on a new FileStorage packer that is 2-3 times
>> faster and, I believe, much less disruptive than the current packing
>> algorithm.
>
> If you are at it: I think the lock which protects the "finish" test
> is hold too long. Currently, it is just release for a very short time
> and then immeadiately reacquired. It should be safe to release it
> immediately after the "finish" test has failed.


I don't know what you are referring to. Could you be more specific?

Note that the packing code in zc.FileStorage releases the commit lock  
far more frequently than the old pack code and doesn't begin acquiring  
it until after copying data written up to the time packing started,  
rather than after writing data up to the pack time.  I expect that the  
packing code in 3.9 will acquire the commit lock much later.

Jim

--
Jim Fulton
Zope Corporation