[ZODB-Dev] ZODB file never updated - import transaction fails...

[skip at pobox.com]

    Chris> The first thing I'd try to do is to move the cleanup code out of
    Chris> __del__ (and perhaps to a finally: after the line that creates
    Chris> the cache instead) and see if that helps.

    Dieter> Avoid using "__del__" and call your close explicitely.

Good point about __del__.  I changed the import of the dnscache code to
this:

    try:
        import dnscache
        cache = dnscache.cache(cachefile=os.path.expanduser("~/.dnscache"))
        cache.printStatsAtEnd = True
    except (IOError, ImportError):
        cache = None
    else:
        import atexit
        atexit.register(cache.close)
        print "calling", cache.close, "at exit"

It dutifully prints "calling <bound method cache.close of
<spambayes.dnscache.cache instance at 0x119a4b8>> at exit" at startup.  I
added a print to the close method.  It prints "closing zodb" at the right
time.  There is no longer any complaint about importing the transaction
module.  As far as I can tell, it should be working.  However, when I look
at the timestamps on the various files, it appears only the .dnscache.index
file has been updated:

    % ls -ltr ~/.dnscache*
    -rw-rw-r--   1 skip  staff  44547 Jul 23 09:06 /Users/skip/.dnscache.old
    -rw-rw-r--   1 skip  staff      0 Jul 23 09:07 /Users/skip/.dnscache.tmp
    -rw-rw-r--   1 skip  staff  44383 Jul 29 08:22 /Users/skip/.dnscache
    -rw-rw-r--   1 skip  staff    293 Jul 29 18:13 /Users/skip/.dnscache.index

(It's about 18:20 here right now.)

So this code appears to be getting called and executes to completion:

    def _zodb_store(self):
        import transaction
        from ZODB.POSException import ConflictError
        from ZODB.POSException import TransactionFailedError

        try:
            transaction.commit()
        except ConflictError, msg:
            # We'll save it next time, or on close.  It'll be lost if we
            # hard-crash, but that's unlikely, and not a particularly big
            # deal.
            if options["globals", "verbose"]:
                print >> sys.stderr, "Conflict on commit.", msg
            transaction.abort()
        except TransactionFailedError, msg:
            # Saving isn't working.  Try to abort, but chances are that
            # restarting is needed.
            if options["globals", "verbose"]:
              print >> sys.stderr, "Store failed.  Need to restart.", msg
            transaction.abort()

    def _zodb_close(self):
        # Ensure that the db is saved before closing.  Alternatively, we
        # could abort any waiting transaction.  We need to do *something*
        # with it, though, or it will be still around after the db is
        # closed and cause problems.  For now, saving seems to make sense
        # (and we can always add abort methods if they are ever needed).
        self._zodb_store()

        # Do the closing.        
        self._DB.close()

        # We don't make any use of the 'undo' capabilities of the
        # FileStorage at the moment, so might as well pack the database
        # each time it is closed, to save as much disk space as possible.
        # Pack it up to where it was 'yesterday'.
        # XXX What is the 'referencesf' parameter for pack()?  It doesn't
        # XXX seem to do anything according to the source.
        self._zodb_storage.pack(time.time()-60*60*24, None)
        self._zodb_storage.close()

        self._zodb_closed = True
        if options["globals", "verbose"]:
            print >> sys.stderr, 'Closed dnscache database'

There are no prints about conflicts or failed stores.  No import errors, the
final "Closed dnscache database" line is output.  Still, there is no update
of the actual data on disk.  Recalling this code:

    from ZODB import DB
    from ZODB.FileStorage import FileStorage
    self._zodb_storage = FileStorage(cachefile, read_only=False)
    self._DB = DB(self._zodb_storage, cache_size=10000)
    self._conn = self._DB.open()
    root = self._conn.root()
    self.caches = root.get("dnscache")
    if self.caches is None:
        # There is no classifier, so create one.
        from BTrees.OOBTree import OOBTree
        self.caches = root["dnscache"] = OOBTree()
        self.caches["A"] = {}
        self.caches["PTR"] = {}

do I need to do something with self.caches or with the root object to
actually flush the data to the file?

One other thing I discovered this morning.  If I delete the ~/.dnscache* and
rerun my SpamBayes training, it obligingly creates a new set of files.  If I
run it again though it doesn't update ~/.dnscache, only ~/.dnscache.tmp and
~/.dnscache.index.

Thx,

-- 
Skip Montanaro - skip at pobox.com - http://www.mojam.com/