[ZODB-Dev] nontransactional, oid, and thread safety. oh my!

[Randy]

Thanks for all of the great information!

Here is the approach I decided to go with.  

I am going to use the FileStorage storage, mechanism.  And, instead of
commiting every change to the database, I am going to do a bunch of
related changes, and them commit them all at once.  (According to the
situation, of course).   This has reduced the size of my database, and
the need to pack as often.

I tried the Berkley storage, but it didn't perform as expected. 
Second, as previously pointed out, the Berkley storage is no longer
supported.  Third, there is no number 3.

Here are the sample Database and Connection abstractions that I'm
using.    The intent is that the Database objects (which are cached in
my code at a higher level) will hand off Connection objects for
incoming (XMLRPC in case you are wondering) requests.

Please don't laugh out loud when you see my code.  I'm still learing.

class Connection:

  """
  A database connection class used to access database data.
  A persistant dictionary is present at Connection.data that can be
used to write data.
  Keys into the dictionary should be strings.
  Use commit() to save your changes, and abort() to rollback your changes.
  CreateId() will create a unique id for you.
  When finished, use Data.close() to close the connection.   Very important!
  """
  def __init__(self, database):
    assert isinstance(database, Database)
    global DATAKEY
    # create a transient connection, that will be closed when this
object is finished
    self.database = database
    self.conn = self.database.db.open()
    root = self.conn.root()
    self.data = root[DATAKEY]

  def __del__(self):
    self.close()

  def __repr__(self):
    return "Connection %s" % self.database.path
  
  def commit(self):
    get_transaction().commit()

  def abort(self):
    get_transaction().abort()

  def createId(self):
    # todo:  this is a potential database "hotspot".. look into alternatives
    global IDKEY
    root = self.conn.root()
    assert root.has_key(IDKEY)
    id = long(root[IDKEY])
    id = id + 1L
    root[IDKEY] = str(id)  # save ID as string.  keys in database are
always strings
    self.commit()    
    return id
  
  def close(self):
    if self.conn:
       print "Closing database connection to %s" % self.database.path
       self.conn.close()
       self.conn = None
       self.data = None

class Database:

    def __init__(self, path):
       self.path = path
       self.storage = ZODB.FileStorage.FileStorage(path)
       self.db = ZODB.DB(self.storage)
       self.packing = False
       self._initialize()

    def __repr__(self):
       return "Database %s" % self.path

    def _initialize(self):
       global DATAKEY, IDKEY
       c = self.db.open()
       root = c.root()
       # high level data table is a btree optimized with string
(object) keys and object data
       if not root.has_key(DATAKEY):
           root[DATAKEY] = BTrees.OOBTree.OOBTree()
       if not root.has_key(IDKEY):
           root[IDKEY] = str(0L)
       get_transaction().commit()  
       c.close()
   
    def connection(self):
       """Get a connection type data object that can be used to write
to the database"""
       return Connection(self)

    def close(self):
       if self.db and self.storage:
           print "Closing database %s" % self.path
           self.db.close()
           self.storage.close()
           self.db = None
           self.storage = None

    def __del__(self):
       self.close()


> [Jim Fulton]
> > Right, but, there are storage implementations that don't require packing.
> >
> > In particular, the Berkeley DB storage doesn't need to be packed at all
> > of you avoid cyclic references among persistent objects and doesn't need
> > to be packed to get rid of old revisions. (Actually, it packs
> > automatically.)  With this storage, you get the benefits of transactions
> > without getting large database growth.  You *do* then need to follow best
> > Berkeley DB practices, like log-file management.
> 
> Toby mentioned BDBStorage too, but I deliberately didn't:  BDB isn't
> supported in the ZODB 3.2 line anymore, and the code for it doesn't even
> exist in ZODB 3.3.  If Randy wants to rehabilitate and take over that code,
> great, but I assumed he was asking for an approach he could "just use".
> 
>