[ZODB-Dev] memory or cache leak?

[Andrew Dalke]

Toby Dickenson <tdickenson@devmail.geminidataloggers.co.uk>:
> Posting rough numbers makes it harder to give a good diagnosis.

Sure.  (Though I did attache the reproducible :)

Numbers based on my post at
  http://lists.zope.org/pipermail/zodb-dev/2001-October/001835.html  

> How many objects have you added to the database at this point?

I've added roughtly 2,400 records.  (The precise number varies
from run to run.)  Each record is called a 'Molecule' and is
defined in 'Database.py' and looks like

class Molecule(Persistence.Persistent):
  def __init__(self, smiles, lookup):
    self._smiles = smiles
    self.lookup = lookup
    self.storage = Persistence.PersistentMapping()
  def __getitem__(self, name):
    return self.storage[name]
  def __setitem__(self, name, value):
    self.storage[name] = value

(Note: Changing PersistentMapping to an OOBTree all through my
doesn't change my results ... although I don't think I've
tested that on my full data run, not this simplified version.  I
started a run like that with my test case while typing the
rest of this email.  The machine I was using ran out of disk space
first, and the process was only 100 MB, but I need to run errands
and don't have the time just now to investigate.)

The ZODB root contains a PersistentMapping term called
"_molecule_lookup" whose key is the molecule's name (called a
'SMILES' and value is this Molecule object).  The SMILES I
use is the string '"mol" + str(i)' where 'i' is the current
index number.

After the Molecule is created, I add two terms to its 'storage'
PersistentMapping member.  These are:
    mol["counter"] = i
    mol["longstring"] = "X" * (128*1024) + str(i)

So a Molecule record adds
  - to _molecule_lookup a new key (string of size a few bytes) and
       value a Molecule
  - Mapping contins a new PersistenceMapping named 'storage', with
       - an integer
       - a string of roughly 128K


> How long does it take?

Oh, about five minute or so.  Depends on the machine configuration
(available RAM, speed of hard disk when hitting virtual memory).
If you look in 'make_db.py' you'll see a line

  #time.sleep(0.1)

This was because I see that one of ZDOB's triggers for a cache
flush occurs by default every minute, and I wanted to make sure
I could give it that time.  (2,000 records * 0.1 sec = 200 sec =
3 minutes > the 1 minute default)

> What is your ZODB configuration (cache size, and time)

Stock.  400 and 60 seconds.

> How big is an object? (256M/4000=65k.... does that sound right?)

Yep.  The major component is the string of size 64k, plus misc.
other data.


> Objects can only be moved out of memory after their state has
> been saved in the database, If the cache size is 4000ish after
> adding 4000 objects then I bet you are not committing them.

Can I take that bet?  I am.  Here's the code from that
'make_db.py' test case I passed around

  if i % 1000 == 0:
    get_transaction().commit()
    db.db.pack(time.time())
    db.db.cacheMinimize(0)
    db.db.cacheFullSweep(0)

I know it's being called because:
  1) I've put print statements there to check
  2) the output (showing the current record number, etc.) pauses for
       a while every 1000 records while it does the commit

                    Andrew
                    dalke@dalkescientific.com