[ZODB-Dev] Relstorage pack problems

[Shane Hathaway]

Santi Camps wrote:
> Sorry, I miss to say we are using Zope 2.9.4, and that mounted object
> is already reachable from the mounted database root (adding a ZODB
> mount point to the root thought ZMI).

Ok.

> Although, the resultant database, after pack, has a big size.  So, not
> all was deleted, but seems the root object not to be found.

A side note: PostgreSQL does not actually reduce the size of a database
until you "vacuum full".  Packing a ZODB in PostgreSQL actually
temporarily increases the size of the database even if it contains a lot
less data.  Still, "vacuum full" is not the right way to control the
database size; the right solution is to enable autovacuum and pack on a
regular basis.

> We are using the same method on a lot of databases and has been
> working fine for a long time, until this case.  I know it's very
> difficult to say what happened, but I've reported the case because
> perhaps it's possible to take some measures in order to avoid data
> loss while packing.  I don't know a lot about ZODB, but
> DirectoryStorage, for instance, don't delete objects immediately, but
> uses a 'deleted' mark and don't remove them until X days (X configured
> in the settings).  I love this caution of DirectoryStorage, but I like
> all the other of RelStorage :-)

Interesting.  RelStorage is already set up to do something like that.
It splits packing into two phases.  In the first phase, it makes a
complete list of every row for the second phase to delete.  The second
phase could be delayed as long as we want.  Between the two phases, if
any database connections use objects that have been marked for deletion,
we could cancel the pack and flag a software bug.

> If you want we make any kind of test with our databases, It will be a pleasure

It would be very helpful to me if you could provide a copy of your
database for me to debug.  I'm hoping for a compressed SQL dump.

Shane