[ZODB-Dev] PopulatingLargeDB

[Tamas Hegedus]

Hi,

I am using ZODB3-3.5.1. I would like to populate it with large number of 
objects (n>100,000). I try to use subtransaction/savepoint to save 
memory after every 10,000th object (my objects have nested objects)

I see the increase of the size of the Data.fs file after commits, but my 
memory does not seem to be freed: the red line just goes upper and upper 
while my disk cash (swap) also started to be used heavily... I have 1G 
RAM...

The size of the file I parse into objects is almost 798M, approx with 
200,000 records.

Parts of the code I use listed below (I tried different combinations of 
commit/savepoint/etc; but I do not really understand the whole picture...).

Thanks for your help and for any suggestion,
Tamas

#==============================================================
db               = ZODB.config.databaseFromURL("etc/zodb.conf")
connection       = db.open()
droot            = connection.root()
droot['MyObjs']  = OOBTree()
myDb             = droot['MyObjs']

#--------------------------------------------------------------
MixIn( MySP.Record, Persistent)

it = MySP.Iterator( open( 'dat_file.txt'),
                      MySP.RecordParser())
i  = 0

for rec in it:
     id       = copy.deepcopy( rec.ids[0])
     myDb[id] = rec
     i += 1
     if i % 10000 == 0:
         transaction.savepoint(True)
	#transaction.commit()
	#transaction.manager.free( transaction.get())
         print i

transaction.commit()
connection.close()
#==============================================================

-- 
Tamas Hegedus, PhD          | phone: (1) 919-966 0329
UNC - Biochem & Biophys     | fax:   (1) 919-966 5178
5007A Thurston-Bowles Bldg  | mailto:hegedus at med.unc.edu
Chapel Hill, NC, 27599-7248 | http://biohegedus.org