[ZODB-Dev] Re: Invalid object references on brand new Data.fs using fsrefs.py

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ross, Troy D wrote:
> Hello,
>   I just created a brand new instance of a Zope 2.9.3, started the
> instance once so that it creates a Data.fs file and then gracefully shut
> down the zope server.  I then ran fsrefs.py on Data.fs and I was given
> the following results:
> 
> oid 0x0L persistent.mapping.PersistentMapping
> last updated: 2006-07-26 21:48:47.904932, tid=0x36714FCCC64F5DDL
> refers to invalid objects:
>         oid ('\x00\x00\x00\x00\x00\x00\x00\x01', None) missing:
> '<unknown>'
>         oid ('\x00\x00\x00\x00\x00\x00\x00\n', None) missing:
> '<unknown>'
> ...
> 
> oid 0xA79L BTrees._IOBTree.IOBTree
> last updated: 2006-07-26 21:48:54.163685, tid=0x36714FCE7193011L
> refers to invalid objects:
>         oid ('\x00\x00\x00\x00\x00\x00\nz', None) missing: '<unknown>'
>         oid ('\x00\x00\x00\x00\x00\x00\n{', None) missing: '<unknown>'
>         oid ('\x00\x00\x00\x00\x00\x00\n|', None) missing: '<unknown>'
> 
> Similar load errors were repeated for a total output of 6713 lines.
>  
> Are these references that don't refer to pickled objects supposed to be
> in the Data.fs file? I've run earlier version of zope and I never saw
> such load errors. The system is running FreeBSD 6.1 with python 2.4.3.
> I tested a similar setup on a Windows XP machine with similar results.

I can reproduce this on the Zope 2.9 branch, but not on the 2.8 branch.

  $ export ZHOME="/path/to/zope"
  $ $ZHOME/bin/mkzopeinstance.py -d /tmp/initrefs -u admin:123
  $ cd /tmp/initrefs
  $ bin/zopectl fg
  ...
  2006-07-24 20:34:27 INFO Zope Ready to handle requests
  ^C
  $ PYTHONPATH="$ZHOME/lib/python" \
    $ZHOME/bin/fsrefs var/Data.fs

My *guess* is that this represents a failure in the fsrefs script, since
the appserver is functioning properly.  Perhaps it has not been updated
to deal with the API changes between ZODB 3.4.x and ZODB 3.6.x?  The
delta between the two is pretty small::

$ svn diff $ZSVN/ZODB/branches/3.{4,6}/src/scripts/fsrefs.py
- --- fsrefs.py   (.../3.4/src/scripts/fsrefs.py) (revision 69267)
+++ fsrefs.py   (.../3.6/src/scripts/fsrefs.py) (revision 69267)
@@ -68,7 +68,8 @@

 from ZODB.FileStorage import FileStorage
 from ZODB.TimeStamp import TimeStamp
- -from ZODB.utils import u64, oid_repr, get_refs, get_pickle_metadata
+from ZODB.utils import u64, oid_repr, get_pickle_metadata
+from ZODB.serialize import get_refs
 from ZODB.POSException import POSKeyError

 VERBOSE = 0
@@ -129,9 +130,8 @@
         refs = get_refs(data)
         missing = [] # contains 3-tuples of oid, klass-metadata, reason
         for info in refs:
- -            try:
- -                ref, klass = info
- -            except (ValueError, TypeError):
+            ref, klass = info
+            if klass is None:
                 # failed to unpack
                 ref = info
                 klass = '<unknown>'


Which does play in the output we see on the 3.6 branch.



Tres.
- --
===================================================================
Tres Seaver          +1 202-558-7113          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEyAzv+gerLs4ltQ4RAtatAKDa2x4rnMpVGxrMHWpb4mEOFcRrCQCdFMBB
XO39Y+Xk11LVLAjjI1/I38M=
=dYiT
-----END PGP SIGNATURE-----