[ZODB-Dev] memory or cache leak?

[Andrew Dalke]

This is a multi-part message in MIME format.

------=_NextPart_000_045B_01C1628A.F2A20D80
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

[Since I haven't heard any response public or private, and since the
original subject was wrong, I'm somewhat repeating my previous email
in the hopes that those in the know find this message more consistent.]

We are running out of memory with our ZODB runs and I can't
figure out why.  Attached are two scripts, 'make_db.py' and a
helper library 'Database.py'.  These are scaled down versions
of our code, and exhibit similar symptoms.

If I run 'make_db.py' the python process grows and grows and
grows.  When we run the full code, it take up a full GB of
RAM.  This smaller version will easily take over 260MB of RAM
(which is how much I gave the process to use).

I can't figure out how to limit the amout of memory used.
I think there's a problem with the caching, as the number of
object in the cache never decreases, and cacheSize() (can get
up to 4,000) goes well beyond the value of getCacheSize()
(set to 400).  The cache doesn't even decrease when I force a
minimizeCache() or do a full sweep.

I tested this under the ZODB/OS combinations of
   2.4.1 / NT SP6
   2.4.1 / Red Hat Linux 7.1
   2.2.2 / NT SP6

They all exhibited the same memory consumption.

I'm not sure what to try next.  I'll try staring at C code,
but barely have an idea of what's going on.  Does anyone
here have any hints?

                    Andrew
                    dalke@dalkescientific.com


------=_NextPart_000_045B_01C1628A.F2A20D80
Content-Type: text/plain;
	name="make_db (1).py"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="make_db (1).py"

import time, sys
import Database

db = Database.create_database("spam.fs")
for i in range(10000):
  print i, db.db.cacheSize()
  sys.stdout.flush()

  mol = db.add("mol" + str(i))
  mol["counter"] = i
  mol["longstring"] = "X" * (128*1024) + str(i)
  if i % 1000 == 0:
    get_transaction().commit()
    db.db.pack(time.time())
    db.db.cacheMinimize(0)
    db.db.cacheFullSweep(0)

  #time.sleep(0.1)

print
get_transaction().commit()
db.db.pack(time.time())
get_transaction().commit()

------=_NextPart_000_045B_01C1628A.F2A20D80
Content-Type: text/plain;
	name="Database (1).py"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="Database (1).py"

import ZODB
from ZODB import FileStorage

import Persistence
from App import Product # hack -- ensures ZODB imports things correctly
from Products.ZCatalog.Catalog import Catalog

class Molecule(Persistence.Persistent):
  def __init__(self, smiles, lookup):
    self._smiles = smiles
    self.lookup = lookup
    self.storage = Persistence.PersistentMapping()
  def __getitem__(self, name):
    return self.storage[name]
  def __setitem__(self, name, value):
    self.storage[name] = value

def create_database(filename):
  db = ZODB.DB(FileStorage.FileStorage(filename))
  connection = db.open()
  root = connection.root()
  root["database_version"] = "1"

  root["molecule_lookup"] = Persistence.PersistentMapping()
  root["storage"] = Persistence.PersistentMapping()

  cat = root["catalog"] = Catalog()
  cat.aq_parent = root

  get_transaction().commit()
  db.close()

  db = Database(filename)
  return db

class Database:
  def __init__(self, filename):
    self.db = ZODB.DB(FileStorage.FileStorage(filename))
    self.connection = self.db.open()
    self.root = self.connection.root()
    self._molecule_lookup = self.root["molecule_lookup"]
    self.storage = self.root["storage"]

  def add(self, smiles):
    mol = Molecule(smiles, self._molecule_lookup)
    self._molecule_lookup[smiles] = mol
    return mol

  def lookup(self, smiles):
    return self._molecule_lookup.lookup(smiles)

  def molecules(self):
    return self._molecule_lookup.values()


------=_NextPart_000_045B_01C1628A.F2A20D80--