[ZODB-Dev] Persistent Mapping Classes

[Chris McDonough]

On Fri, 2005-10-21 at 03:45 -0600, Ron Wills wrote:
> On Thu, 2005-10-20 at 22:16 -0400, Tim Peters wrote:
> > [Ron Wills]
> > >  I hope this is the right place to ask about this...
> > 
> > Yup!  Although given the nearly detail-free scenarios you gave us, I hope
> > you weren't hoping for answers too ;-)
> > 
> 
> Ok, I'll simplify.
> 
> basically the MLDB is:
> 
> class MLDB:
>     def __init__(self, filename):
>         filestore = FileStorage(oodb)
>         database = DB(filestore)
>         conn = DB.open(database)
>         self.db = database
>         self.conn = conn
> 
>     def __del__(self):
>         self.db.close()


"oodb" is a global somewhere we don't know about?

FWIW, overriding __del__ has been a frequent source of problems for me
if I'm not certain that having __del__ called more than once is
idempotent.  I try to not do this, as a result, although it may be
perfectly fine in your application.

> This is a very thin wrapper, which only one instance is created globally

Does your application use multiple threads?

> and all other methods of the class are for our query engine and
> importing routines from a third party sql db. This is not the source of
> the problem and can be removed. So if we take the OORTable class as in
> the previous email (and there is no __del__ method) with a example as
> follows:
> 
> from persistent import Persistent
> from persistent.mapping import PersistentMapping
> 
> class OORTable(PersistentMapping):
>     def __init__(self, base_class):
>         PersistentMapping.__init__(self)
> 
>         self._table_name = name or base_class.__name__
>         self._columns = PersistentMapping()
>         self._record_count = 0
> 
>     del __len__(self):
>         return self._record_count
> 
>     # The other methods are trivial for the example

I hope so, because __init__ can't work as it's written (there doesn't
appear to be a "name" binding, so that bit of code should fail with a
NameError).

> class SomeClass(Persistent):
>     pass
> 
> filestore = FileStorage('myzodb.fs')
> db = DB(filestore)
> conn = DB.open(db)
> 
> conn.root['SomeClass'] = OORTable(SomeClass)
> an_obj = SomeClass()
> an_obj.an_attr = 'a value'
> conn.root['SomeClass']['an_obj'] = an_obj
> 
> db.close()
> 
> This all works as expected, until we reopen the database as I specified
> in the my previous email.

I would expect that, because you didn't use or commit a transaction.
Essentially, on a reopen of the database, you will be using the state
that existed in the database before you did all that (your changes will
just vanish).

> 
> filestore = FileStorage('myzodb.fs')
> db = DB(filestore)
> conn = DB.open(db)
> 
> ootable = conn.root['SomeClass']  # No problem
> an_obj = conn.root['SomeClass']['an_obj'] # No problem
> print an_obj.an_attr # No problem
> 
> len(ootable) # Most of the time raises an Attribute error that 
>              # _record_count doesn't exist. This also happens
>              # with other methods that accesses any other of
>              # the attributes defined.
> 
> db.close()

Can't really guess about the _record_count error, given that you didn't
commit a transaction and we can't know what the state of the database
was before you tried to change it.

> 
>   To hopefully to make this problem clearer any objects we put into
> 'myzodb.fs' that inherit Persistent, or a PersistentMapping(), or a
> OOBTree() end up serialized in 'myzodb.fs'. This one class that inherits
> PersistentMapping, OORTable, is inconstantly serializing its attributes
> to 'myzodb.fs'. Why this is happening or to give consistent testing for
> the failure is beyond me, and unfortunately I cannot freely share the
> code as this is a contracted commercial product :(. But once again,
> here's how it does fail:
> 
> Test Suite:
> 1. Open ZODB.fs & populate
> 2. run tests all pass
> 3. close ZODB.fs, no failures
> 4. reopen ZODB.fs
> 5. run the same tests, fails from missing attributes in OORTable
> 6. close ZODB.fs
> 
> Second Test Suite
> 1. Open ZODB.fs & populate
> 3. close ZODB.fs, no failures
> 4. reopen ZODB.fs
> 5. run tests all pass
> 6. close ZODB.fs, no failures
> 7. repeat from 4 with no problems
> 
> The same sort of bazaar behavior from the web site:
> 
> >>> from ClearWater import MLDB 
> >>> mldb = MLDB.MLDB(_cfg)
> >>> MLDB.mldb = mldb
> >>> # In this case none of the attribute for any OORTable records gets
> >>> # serialized to the ZODB. If shutdown and restarted then 
> >>> # Attribute errors are raised from all the OORTable objects
> 
> >>> from ClearWater import MLDB
> >>> MLDB.mldb = MLDB.MLDB(_cfg)
> >>> # In this case the attributes do get serialize and on a shutdown and
> >>> # restart all the OORTable's attribute exist with proper values.
> 
> Note that all access to the ZODB is done through the global MLDB.mldb
> instance and the minor difference in the above code is the difference
> from a complete or corrupted ZODB.

I don't believe there is any corruption here; if there is, there isn't
any evidence of that.  I do believe that you think the state of the
database isn't what you expect it to be when reopened, but the fact that
there aren't any "import transaction; transaction.commit()" calls or
variants in your code would explain that at large.

>   What is making this even more confusing is it's just the attributes
> from OORTable that are not serialized to the ZODB.fs. All the other
> objects are present, complete and valid. The only thing I'm seeing is
> that PersistentMapping is not consistently serializing it's attributes.
>   This is about the best consistent tests for this problem I can give
> after extensively debugging code and changing the test suites for 3
> days, to narrow down where this problem lies, just short of digging into
> the persistent modules C code.

Try committing the transaction.

- C