[ZODB-Dev] Relstorage and over growing database.

[Jens W. Klein]

On 2013-11-12 17:48, Martijn Pieters wrote:
>
> On Mon, Nov 11, 2013 at 9:24 PM, Daniel Widerin <daniel at widerin.net
> <mailto:daniel at widerin.net>> wrote:
>
>     Anyone experienced similar problems packing large relstorage databases?
>     The graph traversal takes a really long time. maybe we can improve that
>     by storing additional information in the relational database?
>
>
> You should (at least initially) pack *without* GC (set pack-gc true to
> false); I packed a humongous RelStorage-backed database before, and
> packed to earlier dates in the past first to minimize the amount of data
> removed in a single transaction.
>
> Only when we were down to a reasonable size database did we enable
> garbage collection.

Thanks Martijn for the hint, but we are using a history free database, 
so growing does only happen by deleted objects in our case.

When in history free mode, is it possible to detect deleted objects at 
store-time? This way we could add the zoid at store time to a 
objects_deleted table in order to clean them up later.

Another way to speed up graph traversal would be to store the 
object-references in a field of object_state. At the moment we have to 
read the pickle in order to get the referenced zoids. Storing additional 
- redundant - information might be not perfect, but it would allow to 
pack/gc the database without any knowledge about the state objects 
structure, i.e. using a stored procedure.

I would like to know what the relstorage experts think about this ideas.

kind regards
Jens
-- 
Klein & Partner KG, member of BlueDynamics Alliance