[ZODB-Dev] More ZODB POSKeyErrors with 2.6.0

[Guido van Rossum]

> >>>>> "CW" == Chris Withers <chrisw@nipltd.com> writes:
> 
>   CW> I got lots of:
> 
>   CW> ZODB.POSException.POSKeyError: 0000000000082a70
> 
>   CW> ...the OID stayed the same.
> 
>   CW> Solved it by stopping the storage server and deleting all the
>   CW> indexes.  Haven't had problems since the restart.
> 
>   CW> Is there a bug in the FileStorage index lurking here?

[Jeremy]
> Maybe.  I think we need more information to be able to debug the
> problem.  It's hard to guess what would cause an index error.
> 
> It would help if you could run fsrefs.py on the real FileStorage.
> That would tell us if there are any dangling references in the
> storage, or if the problem is solely in the index.
> 
> If it happens again, it would be interesting to keep a copy of the
> index file and see if it is corrupted.

Could the index file have been for a different version of the
database???  E.g. what would happen if the database is updated for a
while, and then the storage server is killed with SIGKILL so the index
is not rewritten?  On storage server restart, an old version of the
index will be found and used, right?  Could that scenario cause this
effect?

CW> PS: Also noticed that I appear to have lost tracebacks on this
CW> now, I wonder why that could be?

Do you mean that in the log file, you see a message with level ERROR
but not followed by a traceback?  We've seen this a few times at a
customer site.  AFAIK we don't know yet how this can happen, but the
code in lib/python/zExceptions/ExceptionFormatter.py is suspect.
Maybe zLOG should *not* try to use this (see
lib/python/zLOG/FormatException.py) but always fall back to the
standard traceback module's format_exception?

--Guido van Rossum (home page: http://www.python.org/~guido/)