[ZODB-Dev] Persistent Mapping Classes

[Ron Wills]

On Thu, 2005-10-20 at 22:16 -0400, Tim Peters wrote:
> [Ron Wills]
> >  I hope this is the right place to ask about this...
> 
> Yup!  Although given the nearly detail-free scenarios you gave us, I hope
> you weren't hoping for answers too ;-)
> 

Ok, I'll simplify.

basically the MLDB is:

class MLDB:
    def __init__(self, filename):
        filestore = FileStorage(oodb)
        database = DB(filestore)
        conn = DB.open(database)
        self.db = database
        self.conn = conn

    def __del__(self):
        self.db.close()

This is a very thin wrapper, which only one instance is created globally
and all other methods of the class are for our query engine and
importing routines from a third party sql db. This is not the source of
the problem and can be removed. So if we take the OORTable class as in
the previous email (and there is no __del__ method) with a example as
follows:

from persistent import Persistent
from persistent.mapping import PersistentMapping

class OORTable(PersistentMapping):
    def __init__(self, base_class):
        PersistentMapping.__init__(self)

        self._table_name = name or base_class.__name__
        self._columns = PersistentMapping()
        self._record_count = 0

    del __len__(self):
        return self._record_count

    # The other methods are trivial for the example

class SomeClass(Persistent):
    pass

filestore = FileStorage('myzodb.fs')
db = DB(filestore)
conn = DB.open(db)

conn.root['SomeClass'] = OORTable(SomeClass)
an_obj = SomeClass()
an_obj.an_attr = 'a value'
conn.root['SomeClass']['an_obj'] = an_obj

db.close()

This all works as expected, until we reopen the database as I specified
in the my previous email.

filestore = FileStorage('myzodb.fs')
db = DB(filestore)
conn = DB.open(db)

ootable = conn.root['SomeClass']  # No problem
an_obj = conn.root['SomeClass']['an_obj'] # No problem
print an_obj.an_attr # No problem

len(ootable) # Most of the time raises an Attribute error that 
             # _record_count doesn't exist. This also happens
             # with other methods that accesses any other of
             # the attributes defined.

db.close()

  To hopefully to make this problem clearer any objects we put into
'myzodb.fs' that inherit Persistent, or a PersistentMapping(), or a
OOBTree() end up serialized in 'myzodb.fs'. This one class that inherits
PersistentMapping, OORTable, is inconstantly serializing its attributes
to 'myzodb.fs'. Why this is happening or to give consistent testing for
the failure is beyond me, and unfortunately I cannot freely share the
code as this is a contracted commercial product :(. But once again,
here's how it does fail:

Test Suite:
1. Open ZODB.fs & populate
2. run tests all pass
3. close ZODB.fs, no failures
4. reopen ZODB.fs
5. run the same tests, fails from missing attributes in OORTable
6. close ZODB.fs

Second Test Suite
1. Open ZODB.fs & populate
3. close ZODB.fs, no failures
4. reopen ZODB.fs
5. run tests all pass
6. close ZODB.fs, no failures
7. repeat from 4 with no problems

The same sort of bazaar behavior from the web site:

>>> from ClearWater import MLDB 
>>> mldb = MLDB.MLDB(_cfg)
>>> MLDB.mldb = mldb
>>> # In this case none of the attribute for any OORTable records gets
>>> # serialized to the ZODB. If shutdown and restarted then 
>>> # Attribute errors are raised from all the OORTable objects

>>> from ClearWater import MLDB
>>> MLDB.mldb = MLDB.MLDB(_cfg)
>>> # In this case the attributes do get serialize and on a shutdown and
>>> # restart all the OORTable's attribute exist with proper values.

Note that all access to the ZODB is done through the global MLDB.mldb
instance and the minor difference in the above code is the difference
from a complete or corrupted ZODB.

  What is making this even more confusing is it's just the attributes
from OORTable that are not serialized to the ZODB.fs. All the other
objects are present, complete and valid. The only thing I'm seeing is
that PersistentMapping is not consistently serializing it's attributes.
  This is about the best consistent tests for this problem I can give
after extensively debugging code and changing the test suites for 3
days, to narrow down where this problem lies, just short of digging into
the persistent modules C code.

> 
-- 
Ron Wills <ron at rwsoft.ca>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20051021/dd80eb0d/attachment.bin