Hi Jim,<div><br></div><div>I'm really sorry for the miscommunication, I thought I made that clear in my last email:</div><div><br></div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 12.5px; border-collapse: collapse; ">"I'm wrapping ZODB in a 'ZMap' class that just forwards all the dictionary methods to the ZODB root and allows easy interchangeability with my old sqlite OODB abstraction."</span></div>
<div><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 12.5px; border-collapse: collapse; "><br></span></div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; border-collapse: collapse; ">wordid_to_docset is a "ZMap", which just wraps the ZODB boilerplate/connection and forwards dictionary methods to the root. If this seems superfluous, it was just to maintain backwards compatibility with all of the code I'd already written for the sqlite OODB I was using before I switched to ZODB. Whenever you see something like wordid_to_docset[id] it's just doing self.root[id] behind the scenes in a __setitem__ call inside the ZMap class, which I've pasted below.</span></div>
<div><span class="Apple-style-span" style="font-family: arial, sans-serif; border-collapse: collapse; "><br></span></div><div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">The db is just storing longs mapped to array('L')'s with a few thousand longs in em. I'm going to try switching to the persistent data structure that Laurence suggested (a pointer to relevant documentation would be really useful), but I'm still sorta worried because in my experimentation with ZODB so far I've never been able to observe it sticking to any cache limits, no matter how often I tell it to garbage collect (even when storing very small values that should give it adequate granularity...see my experiment at the end of my last email). If the memory reported to the OS by Python 2.6 is the problem I'd understand, but memory usage goes up the second I start adding new things (which indicates that Python is asking for more and not actually freeing internally, no?).</span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;"><br></span></font></div><div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">If you feel there's something pathological about my memory access patterns in this operation I can just do the actual inversion step in Hadoop and load the output into ZODB for my application later, I was just hoping to keep all of my data in OODB's the entire time.</span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;"><br></span></font></div><div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">Thanks again all of you for your collective time. I really like ZODB so far, and it bugs me that I'm likely screwing it up somewhere.</span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;"><br></span></font></div><div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">Cheers,</span></font></div>
<div><font class="Apple-style-span" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse;">Ryan</span></font></div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 12.5px; border-collapse: collapse; "><br>
</span></div><div><font class="Apple-style-span" face="arial, sans-serif" size="4"><span class="Apple-style-span" style="border-collapse: collapse; font-size: 15px;"><br></span></font></div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 12.5px; border-collapse: collapse; "><br>
</span></div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 12.5px; border-collapse: collapse; "><div><font class="Apple-style-span" face="'courier new', monospace"><div>class ZMap(object):</div>
<div> </div><div> def __init__(self, name=None, dbfile=None, cache_size_mb=512, autocommit=True):</div><div> <a href="http://self.name">self.name</a> = name</div><div> self.dbfile = dbfile</div><div> self.autocommit = autocommit</div>
<div> </div><div> self.__hash__ = None #can't hash this</div><div> </div><div> #first things first, figure out if we need to make up a name</div><div> if <a href="http://self.name">self.name</a> == None:</div>
<div> <a href="http://self.name">self.name</a> = make_up_name()</div><div> if sep in <a href="http://self.name">self.name</a>:</div><div> if <a href="http://self.name">self.name</a>[-1] == sep:</div>
<div> <a href="http://self.name">self.name</a> = <a href="http://self.name">self.name</a>[:-1]</div><div> <a href="http://self.name">self.name</a> = self.name.split(sep)[-1]</div><div> </div>
<div> </div><div> if self.dbfile == None:</div><div> self.dbfile = <a href="http://self.name">self.name</a> + '.zdb'</div><div> </div><div> self.storage = FileStorage(self.dbfile, pack_keep_old=False)</div>
<div> self.cache_size = cache_size_mb * 1024 * 1024</div><div> </div><div> self.db = DB(self.storage, pool_size=1, cache_size_bytes=self.cache_size, historical_cache_size_bytes=self.cache_size, database_name=<a href="http://self.name">self.name</a>)</div>
<div> self.connection = self.db.open()</div><div> self.root = self.connection.root()</div><div> </div><div> print 'Initializing ZMap "%s" in file "%s" with %dmb cache. Current %d items' % (<a href="http://self.name">self.name</a>, self.dbfile, cache_size_mb, len(self.root))</div>
<div><br></div><div> </div><div><br></div><div> # basic operators</div><div> def __eq__(self, y): # x == y</div><div> return self.root.__eq__(y)</div><div> def __ge__(self, y): # x >= y</div><div>
return len(self) >= len(y)</div><div> def __gt__(self, y): # x > y</div><div> return len(self) > len(y)</div><div> def __le__(self, y): # x <= y</div><div> return not self.__gt__(y)</div>
<div> def __lt__(self, y): # x < y</div><div> return not self.__ge__(y)</div><div> def __len__(self): # len(x)</div><div> return len(self.root)</div><div> </div><div> </div><div> # dictionary stuff </div>
<div> def __getitem__(self, key): # x[key]</div><div> return self.root[key]</div><div><br></div><div> def __setitem__(self, key, value): # x[key] = value</div><div> self.root[key] = value</div><div> self.__commit_check() # write back if necessary </div>
<div> </div><div> def __delitem__(self, key): # del x[key]</div><div> del self.root[key]</div><div> </div><div><br></div><div> def get(self, key, default=None): # x[key] if key in x, else default</div>
<div> return self.root.get(key, default)</div><div><br></div><div> def has_key(self, key): # True if x has key, else False</div><div> return self.root.has_key(key)</div><div><br></div><div> def items(self): # list of key/val pairs</div>
<div> return self.root.items()</div><div><br></div><div> def keys(self):</div><div> return self.root.keys()</div><div><br></div><div> def pop(self, key, default=None):</div><div> return self.root.pop()</div>
<div><br></div><div> def popitem(self): #remove and return an arbitrary key/val pair</div><div> return self.root.popitem()</div><div><br></div><div> def setdefault(self, key, default=None):</div><div> #D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D</div>
<div> return self.root.setdefault(key, default)</div><div><br></div><div> def values(self):</div><div> return self.root.values()</div><div> </div><div> def copy(self): #copy it? dubiously necessary at the moment</div>
<div> NOT_IMPLEMENTED('copy')</div><div> </div><div> </div><div> # iteration</div><div> def __iter__(self): # iter(x)</div><div> return self.root.iterkeys()</div><div> </div>
<div> def iteritems(self): #iterator over items, this can be hellaoptimized</div><div> return self.root.iteritems()</div><div><br></div><div><br></div><div> def itervalues(self):</div><div> return self.root.itervalues()</div>
<div><br></div><div> def iterkeys(self):</div><div> return self.root.iterkeys()</div><div> </div><div><br></div><div> # practical realities of the abstraction</div><div> def garbage_collect(self):</div>
<div> self.root._p_jar.cacheGC()</div><div> #self.connection.cacheGC()</div><div> </div><div> def commit(self):</div><div> return self.__commit_check(force=True)</div><div> </div><div> def __commit_check(self, force=False):</div>
<div> if self.autocommit or force:</div><div> transaction.commit()</div></font></div><div><br></div></span></div><div><span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 12.5px; border-collapse: collapse; "><br>
</span></div><div><br><div class="gmail_quote">On Tue, May 11, 2010 at 3:50 AM, Jim Fulton <span dir="ltr"><<a href="mailto:jim@zope.com">jim@zope.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Mon, May 10, 2010 at 8:20 PM, Ryan Noon <<a href="mailto:rmnoon@gmail.com">rmnoon@gmail.com</a>> wrote:<br>
> P.S. About the data structures:<br>
> wordset is a freshly unpickled python set from my old sqlite oodb thingy.<br>
> The new docsets I'm keeping are 'L' arrays from the stdlib array module.<br>
> I'm up for using ZODB's builtin persistent data structures if it makes a<br>
> lot of sense to do so, but it sorta breaks my abstraction a bit and I feel<br>
> like the memory issues I'm having are somewhat independent of the container<br>
> data structures (as I'm having the same issue just with fixed size strings).<br>
<br>
</div>This is getting tiresome. We can't really advise you because we can't<br>
see what data structures you're using and we're wasting too much time<br>
guessing. We wouldn't have to guess and grill you if you showed a<br>
complete demonstration program, or at least one that showed what the<br>
heck your doing.<br>
<br>
The program you've showed so far is so incomplete, perhaps we're<br>
missing the obvious.<br>
<br>
In your original program, you never actually store anything in the<br>
database. You assign the database root to self.root, but never use<br>
self.root. (The variable self is not defined and we're left to assume<br>
that this disembodied code is part of a method definition.) In your<br>
most recent snippet, you don't show any database access. If you<br>
never actually store anything in the database, then nothing will be<br>
removed from memory.<br>
<br>
You're inserting data into wordid_to_docset, but you don't show its<br>
definition and won't tell us what it is.<br>
<br>
Jim<br>
<br>
--<br>
<font color="#888888">Jim Fulton<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>Ryan Noon<br>Stanford Computer Science<br>BS '09, MS '10<br>
</div>