<span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 13px; border-collapse: collapse; ">Hi everyone (I just registered for the list and resent so as to not bug the mod),<div><br></div><div>I recently switched over some of my home-rolled sqlite backed object databases into ZODB based on what I'd read and some cool performance numbers I'd seen. I'm really happy with the entire system so far except for one really irritating problem: memory usage.</div>
<div><br></div><div>I'm doing a rather intensive operation where I'm inverting a mapping of the form (docid => [wordid]) for about 3 million documents (for about 8 million unique words). I thought about doing it on hadoop, but it's a one time thing and it'd be nice if I didn't have to load the data back into an object database for my application at the end anyway.</div>
<div><br></div><div>Anyhoo, in the process of this operation (which performs much faster than my sqlite+python cache solution) memory usage never really drops. I'm currently doing a commit every 25k documents. The python process just gobbles up RAM, though. I made it through 750k documents before my 8GB Ubuntu 10.04 server choked and killed the process (at about 80 percent mem usage). (The same thing happens on Windows and OSX, btw).</div>
<div><br></div><div>I figure either there's a really tremendous bug in ZODB (unlikely given its age and venerability) or I'm really doing it wrong. Here's my code:</div><div><br></div><div><br></div><div><div>
self.storage = FileStorage(self.dbfile, pack_keep_old=False)</div><div> cache_size = 512 * 1024 * 1024</div><div> </div><div> self.db = DB(self.storage, pool_size=1, cache_size_bytes=cache_size, historical_cache_size_bytes=cache_size, database_name=<a href="http://self.name/" target="_blank" style="color: rgb(6, 88, 181); ">self.name</a>)</div>
<div> self.connection = self.db.open()</div><div> self.root = self.connection.root()</div><div><br></div><div><br></div><div>and the actual insertions...</div><div><br></div><div><div> set_default = wordid_to_docset.root.setdefault #i can be kinda pathological with loop operations</div>
<div> array_append = array.append</div><div> for docid, wordset in docid_to_wordset.iteritems(): #one of my older sqlite oodb's, not maintaining a cache...just iterating (small constant mem usage)</div>
<div> for wordid in wordset:</div><div> docset = set_default(wordid, array('L'))</div><div> array_append(docset, docid)</div><div> </div><div> n_docs_traversed += 1</div>
<div> if n_docs_traversed % 1000 == 1:</div><div> status_tick()</div><div> if n_docs_traversed % 25000 == 1:</div><div> self.do_commit() #just commits the oodb by calling transaction.commit()</div>
</div><div><br></div><div>The DB on the choked process is perfectly good up to the last commit when it choked, and I've even tried extremely small values of cache_size_bytes and cache_size, just to see if I can get it to stop allocating memory and nothing seems to work. I've also used string values ('128mb') for cache-size-bytes, etc.</div>
<div><br></div><div><br></div><div>Can somebody help me out?</div><div><br></div><div>Thanks,</div><div>Ryan</div></div></span><br>-- <br>Ryan Noon<br>Stanford Computer Science<br>BS '09, MS '10<br>