[ZODB-Dev] POSKeyError plea for help

[Toby Dickenson]

On Monday 23 December 2002 6:21 pm, Chris Withers wrote:
> Toby Dickenson wrote:
> > On Monday 23 December 2002 3:02 pm, Chris Withers wrote:
> >>If anyone can help with this:
> >>http://sourceforge.net/tracker/index.php?func=3Ddetail&aid=3D652618&g=
roup_id=3D
> >>15 628&atid=3D115628
> >>
> >>...I'd really appreciate it, it's starting to become a major problem =
on a
> >>customer site :-(
> >
> > Jeremy suggested you try the fsrefs script to see if there really are=
 any
> > missing objects. I dont remember seeing your results?
>
> http://lists.zope.org/pipermail/zodb-dev/2002-December/004104.html
>
> My general reading is that there aren't any missing objects...

That has confirmed that two storages are broadly ok. fsrefs uses the norm=
al=20
storage code to parse the storage files, so it is not a very thorough tes=
t.=20
(I could hypothesize about some corruption in your storage that would cau=
se=20
fsrefs to detect a problem one day but not another, which would be consis=
tent=20
with your original problem)

It has also confirmed that one of your storages has at least one missing=20
object. It may have more. If your live exceptions are from that storage t=
hen=20
I think you have at least part of your answer..... but we still dont know=
=20
what caused this corruption.

> How would I copy everything from a FileStorage to a DirStorage?

An ordinary copyTransactionsFrom script... example included below.=20

> What level of support do you offer for DirStorage should other
> bad things start happening?

I am using it in production now, so I am committed to maintaining it.=20

On Monday 23 December 2002 6:40 pm, Barry A. Warsaw wrote:
> You could also try BDBFull storage.  Part of my job is to make sure it
> works correctly!

I designed DirectoryStorage in response to my storages having problems mu=
ch=20
like Chris is reporting, so for this particular problem I think there are=
=20
reasons to prefer DirectoryStorage:

(I hope Barry thinks this is a fair comparison)

1. It performs dangling reference checking. If your corruption is being c=
aused=20
by an application or zope bug then DirectoryStorage is the only storage t=
hat=20
will prevent the corruption, by detecting the bad transactions before the=
y=20
are committed.

2. It has a checking tool, so you can detect any other problems early.
(run this immediately after the copyTransactionsFrom script too)

3. There are some known corner cases to the packing algorithm used by all=
=20
storages which can cause permanent storage corruption. It is generally ag=
reed=20
that they should be hit very rarely, possibly only when the hole is expos=
ed=20
by an application bug. (DirectoryStorage and BDBAutopack have tweaks to=20
further reduce this hole). DirectoryStorage is the only storage that will=
 let=20
you recover from this corruption by reversing the packing process. (disk=20
space is not reclaimed until a few days after a pack to allow this - so m=
ake=20
sure you run the checking tool often enough to make best use of this=20
protection)=20

4. Assuming you are not already a berkelydb guru, DirectoryStorage has a =
lower=20
learning curve to climb before you can manage it safely without risking=20
self-inflicted damage to your storage. I last used bsdbStorage.Full 10 mo=
nths=20
ago, and at that time I managed to destroy several storages before I=20
understood the subtleties of the backup procedure described around here:
http://www.sleepycat.com/docs/ref/transapp/archival.html
http://www.sleepycat.com/docs/ref/transapp/reclimit.html

5. If you do encounter problems DirectoryStorage should make it easier fo=
r you=20
to perform first line debugging, if necessary, by looking at the raw stor=
age=20
files. There is the beginings of a technical reference at=20
http://dirstorage.sourceforge.net/technical.html, but the directory struc=
ture=20
is fairly simple so you should be able to dive in using the dscat.py tool=
 (to=20
dump the content of individual storage files), find, and ls.

--

import sys

sys.path.append('/home/tdickenson/projects/DCZope/Working/lib/python')

from ZODB.FileStorage import FileStorage

from DirectoryStorage.Full import Full as DS
from DirectoryStorage.Filesystem import Filesystem

src =3D FileStorage('/home/tdickenson/calarchive2.fs',read_only=3D1)

fs =3D Filesystem('/home/tdickenson/big/directorystorages/w1')
dst =3D DS(fs)

dst.copyTransactionsFrom(src)

src.close()
dst.close()