[ZODB-Dev] Optimizing RelStorage Packing for large DBs

[Jens W. Klein]

On 2013-11-18 17:29, Shane Hathaway wrote:
> On 11/15/2013 06:01 PM, Jens W. Klein wrote:
>> The idea is simple:
>>
>> - iterate over all transactions starting with the lowest
>>    transaction id (tid)
>> - for each transaction load the object states connected with tid
>> - for each state fetch its outgoing references and fill a table where
>>    all incoming references of an object are stored as an array.
>>    if an state has no references write it anyway to the table with empty
>>    outgoing references
>
> I would describe the RelStorage packing algorithm with the same words,
> but since you reimplemented the algorithm from scratch, you found a more
> optimal implementation for your database. Good work!
>

Thanks for the feedback!

The only change I made is regarding to arrays - they are difficult to 
handle in postgres (no fun at all). It is much easier (and faster!) to 
add one row for each incoming reference (plus a reference to self). 
Deleting is now easy by fetching all references with only the 
self-reference and removing the objects, its left self-reference and 
deleting incoming references of the objects referenced by the deleted 
object.

My SQL was a bit rusty after zome years of ODBs. I'am sure a person with 
deeper SQL knowledge may optimize the queries in some way.

regards Jens
-- 
Klein & Partner KG, member of BlueDynamics Alliance