[ZODB-Dev] Optimizing RelStorage Packing for large DBs

[Jim Fulton]

On Mon, Nov 18, 2013 at 8:43 AM, Jan-Wijbrand Kolman
<janwijbrand at gmail.com> wrote:
> On 11/18/13 12:19 PM, Jim Fulton wrote:
>>
>> On Fri, Nov 15, 2013 at 8:01 PM, Jens W. Klein <jens at bluedynamics.com>
>> wrote:
>>>
>>> I started a new packing script for Relstorage (history free, postgresql).
>>> It
>>> is based on incoming reference counting.
>>
>>
>> Did you look at zc.zodbdgc?  I think it implements something very close to
>> what you're proposing.  It's been in production for a few years now at ZC.
>>
>> Not sure if it would need to be updated for relstorage.
>
>
> AFAICT it does not work against a relstorage backend. Or at least I think to
> understand that from:
>
> http://www.zodb.org/en/latest/documentation/articles/multi-zodb-gc.html
>
> [...This documentation does not apply to RelStorage which has the same
> features built-in, but accessible in different ways. Look at the options for
> the zodbpack script. The –prepack option creates a table containing the same
> information as we are creating in the reference database[...]

I didn't write that.  I think zodbdgz probably would work, possibly
with some modifications.
If nothing else, it should be consulted, but then again, writing
software is fun.

Note that the important aspect here isn't cross-database references,
but the garbage
collection algorithm, which is incremental and uses a linear scan of
the database.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton