[ZODB-Dev] Relstorage and over growing database.

[Daniel Widerin]

Hi, just want to share our experience:

My ZODB contains 300mio objects on relstorage/pgsql. The amount of
objects is caused by btrees stored on plone dexterity contenttypes. It's
size is 160GB. At that size it's impossible to pack because the pre-pack
takes >100 days.

jensens and me are searching for different packing algorithms and
methods to achieve better packing performance. We're keeping you updated
here!

How i solved my problem for now:

I converted into FileStorage which took about 40 hours and Data.fs was
55GB in size. Now i tried to run zeopack on that database - which
succeeded and database was reduced to 7.8 GB - still containing 40mio
objects. After that i migrated back to relstorage because of better
performance and the result is a 11 GB db in pgsql.

Anyone experienced similar problems packing large relstorage databases?
The graph traversal takes a really long time. maybe we can improve that
by storing additional information in the relational database?

Any hints or comments are welcome.

Daniel

Am 07.02.13 23:10, schrieb Shane Hathaway:
> On 02/07/2013 01:54 PM, Jürgen Herrmann wrote:
>> Am 07.02.2013 21:18, schrieb Jürgen Herrmann:
>>> I know that's entirely not your fault but may be worth mentioning
>>> in the docs. Relstorage with MySQL works *very* well for DB sizes
>>> <5GB or so, above that - not so much :/
>>
>> Also for the docs: on disk Restorage/MySQL uses 4x the size of a
>> FileStorage with same contents. As packing tables are filled this
>> grows by another factor of ~2. If you don't pack very regularly
>> you might up ending in DBs that donb't permit packing anymore
>> because of the big size very quickly.
> 
> I suspect there are ways to fix all of that in the MySQL configuration.
> Like any SQL database, MySQL needs tuning as it grows.  Meanwhile,
> FileStorage doesn't really have any knobs, and it always stores in a
> fairly optimal way, so it's easier to use.
> 
> FileStorage has a couple of issues that often drive people to
> RelStorage: (1) the on-disk format is unique to FileStorage, and there
> aren't many tools available for analyzing and fixing a broken Data.fs.
> (2) FileStorage only supports multiple clients through ZEO, which has
> relatively high latency.  If these issues don't impact you, then
> FileStorage is clearly the better choice for you.
> 
> Shane
> 
> _______________________________________________
> For more information about ZODB, see http://zodb.org/
> 
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev