[ZODB-Dev] PopulatingLargeDB

[Tamas Hegedus]

Hi,

I solved the problem:
MySP.Record was a 3rd party object w/o Persistance. I did not want to 
modify it, therefore I MixedIn the Persistance class into the Record 
class. But not everything...

So, finally, I 'hacked' the module holding Record class (simply giving 
Persistance class as a its base class) and now everything seems to  OK 
memory consuption stopped around 300M. I over the population of the 
100,000th object.

Regards,
Tamas

Tamas Hegedus wrote:
> Hi,
> 
> I am using ZODB3-3.5.1. I would like to populate it with large number of 
> objects (n>100,000). I try to use subtransaction/savepoint to save 
> memory after every 10,000th object (my objects have nested objects)
> 
> I see the increase of the size of the Data.fs file after commits, but my 
> memory does not seem to be freed: the red line just goes upper and upper 
> while my disk cash (swap) also started to be used heavily... I have 1G 
> RAM...
> 
> The size of the file I parse into objects is almost 798M, approx with 
> 200,000 records.
> 
> Parts of the code I use listed below (I tried different combinations of 
> commit/savepoint/etc; but I do not really understand the whole picture...).
> 
> Thanks for your help and for any suggestion,
> Tamas
> 
> #==============================================================
> db               = ZODB.config.databaseFromURL("etc/zodb.conf")
> connection       = db.open()
> droot            = connection.root()
> droot['MyObjs']  = OOBTree()
> myDb             = droot['MyObjs']
> 
> #--------------------------------------------------------------
> MixIn( MySP.Record, Persistent)
> 
> it = MySP.Iterator( open( 'dat_file.txt'),
>                      MySP.RecordParser())
> i  = 0
> 
> for rec in it:
>     id       = copy.deepcopy( rec.ids[0])
>     myDb[id] = rec
>     i += 1
>     if i % 10000 == 0:
>         transaction.savepoint(True)
>     #transaction.commit()
>     #transaction.manager.free( transaction.get())
>         print i
> 
> transaction.commit()
> connection.close()
> #==============================================================
> 

-- 
Tamas Hegedus, PhD          | phone: (1) 919-966 0329
UNC - Biochem & Biophys     | fax:   (1) 919-966 5178
5007A Thurston-Bowles Bldg  | mailto:hegedus at med.unc.edu
Chapel Hill, NC, 27599-7248 | http://biohegedus.org