Thanks Laurence, this looks really helpful. The simplicity of ZODB's concept and the joy of using it apparently hides some of the complexity necessary to use it efficiently. I'll check this out when I circle back to data stuff tomorrow.<div>
<br></div><div>Have a great morning/day/evening!</div><div>-Ryan<br><br><div class="gmail_quote">On Tue, May 11, 2010 at 5:44 PM, Laurence Rowe <span dir="ltr"><<a href="mailto:l@lrowe.co.uk">l@lrowe.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">I think this means that you are storing all of your data in a single<br>
persistent object, the database root PersistentMapping. You need to<br>
break up your data into persistent objects (instances of objects that<br>
inherit from persistent.Persistent) for the ZODB to have a chance of<br>
performing memory mapping. You want to do something like:<br>
<br>
import transaction<br>
from ZODB import FileStorage, DB<br>
from BTrees.LOBTree import BTree, TreeSet<br>
storage = FileStorage.FileStorage('/tmp/test-filestorage.fs')<br>
db = DB(storage)<br>
conn = db.open()<br>
root = conn.root()<br>
transaction.begin()<br>
index = root['index'] = BTree()<br>
values = index[1] = TreeSet()<br>
values.add(42)<br>
transaction.commit()<br>
<br>
You should probably read:<br>
<a href="http://www.zodb.org/documentation/guide/modules.html#btrees-package" target="_blank">http://www.zodb.org/documentation/guide/modules.html#btrees-package</a>.<br>
Since that was written an L variants of the BTree types have been<br>
introduced for storing 64bit integers. I'm using an LOBTree because<br>
that maps 64bit integers to python objects. For values I'm using an<br>
LOTreeSet, though you could also use an LLTreeSet (which has larger<br>
buckets).<br>
<font color="#888888"><br>
Laurence<br>
</font><div><div></div><div class="h5"><br>
On 12 May 2010 00:37, Ryan Noon <<a href="mailto:rmnoon@gmail.com">rmnoon@gmail.com</a>> wrote:<br>
> Hi Jim,<br>
> I'm really sorry for the miscommunication, I thought I made that clear in my<br>
> last email:<br>
> "I'm wrapping ZODB in a 'ZMap' class that just forwards all the dictionary<br>
> methods to the ZODB root and allows easy interchangeability with my old<br>
> sqlite OODB abstraction."<br>
> wordid_to_docset is a "ZMap", which just wraps the ZODB<br>
> boilerplate/connection and forwards dictionary methods to the root. If this<br>
> seems superfluous, it was just to maintain backwards compatibility with all<br>
> of the code I'd already written for the sqlite OODB I was using before I<br>
> switched to ZODB. Whenever you see something like wordid_to_docset[id] it's<br>
> just doing self.root[id] behind the scenes in a __setitem__ call inside the<br>
> ZMap class, which I've pasted below.<br>
> The db is just storing longs mapped to array('L')'s with a few thousand<br>
> longs in em. I'm going to try switching to the persistent data structure<br>
> that Laurence suggested (a pointer to relevant documentation would be really<br>
> useful), but I'm still sorta worried because in my experimentation with ZODB<br>
> so far I've never been able to observe it sticking to any cache limits, no<br>
> matter how often I tell it to garbage collect (even when storing very small<br>
> values that should give it adequate granularity...see my experiment at the<br>
> end of my last email). If the memory reported to the OS by Python 2.6 is<br>
> the problem I'd understand, but memory usage goes up the second I start<br>
> adding new things (which indicates that Python is asking for more and not<br>
> actually freeing internally, no?).<br>
> If you feel there's something pathological about my memory access patterns<br>
> in this operation I can just do the actual inversion step in Hadoop and load<br>
> the output into ZODB for my application later, I was just hoping to keep all<br>
> of my data in OODB's the entire time.<br>
> Thanks again all of you for your collective time. I really like ZODB so<br>
> far, and it bugs me that I'm likely screwing it up somewhere.<br>
> Cheers,<br>
> Ryan<br>
><br>
><br>
> class ZMap(object):<br>
><br>
> def __init__(self, name=None, dbfile=None, cache_size_mb=512,<br>
> autocommit=True):<br>
> <a href="http://self.name" target="_blank">self.name</a> = name<br>
> self.dbfile = dbfile<br>
> self.autocommit = autocommit<br>
><br>
> self.__hash__ = None #can't hash this<br>
><br>
> #first things first, figure out if we need to make up a name<br>
> if <a href="http://self.name" target="_blank">self.name</a> == None:<br>
> <a href="http://self.name" target="_blank">self.name</a> = make_up_name()<br>
> if sep in <a href="http://self.name" target="_blank">self.name</a>:<br>
> if <a href="http://self.name" target="_blank">self.name</a>[-1] == sep:<br>
> <a href="http://self.name" target="_blank">self.name</a> = <a href="http://self.name" target="_blank">self.name</a>[:-1]<br>
> <a href="http://self.name" target="_blank">self.name</a> = self.name.split(sep)[-1]<br>
><br>
><br>
> if self.dbfile == None:<br>
> self.dbfile = <a href="http://self.name" target="_blank">self.name</a> + '.zdb'<br>
><br>
> self.storage = FileStorage(self.dbfile, pack_keep_old=False)<br>
> self.cache_size = cache_size_mb * 1024 * 1024<br>
><br>
> self.db = DB(self.storage, pool_size=1,<br>
> cache_size_bytes=self.cache_size,<br>
> historical_cache_size_bytes=self.cache_size, database_name=<a href="http://self.name" target="_blank">self.name</a>)<br>
> self.connection = self.db.open()<br>
> self.root = self.connection.root()<br>
><br>
> print 'Initializing ZMap "%s" in file "%s" with %dmb cache. Current<br>
> %d items' % (<a href="http://self.name" target="_blank">self.name</a>, self.dbfile, cache_size_mb, len(self.root))<br>
><br>
> # basic operators<br>
> def __eq__(self, y): # x == y<br>
> return self.root.__eq__(y)<br>
> def __ge__(self, y): # x >= y<br>
> return len(self) >= len(y)<br>
> def __gt__(self, y): # x > y<br>
> return len(self) > len(y)<br>
> def __le__(self, y): # x <= y<br>
> return not self.__gt__(y)<br>
> def __lt__(self, y): # x < y<br>
> return not self.__ge__(y)<br>
> def __len__(self): # len(x)<br>
> return len(self.root)<br>
><br>
><br>
> # dictionary stuff<br>
> def __getitem__(self, key): # x[key]<br>
> return self.root[key]<br>
> def __setitem__(self, key, value): # x[key] = value<br>
> self.root[key] = value<br>
> self.__commit_check() # write back if necessary<br>
><br>
> def __delitem__(self, key): # del x[key]<br>
> del self.root[key]<br>
><br>
> def get(self, key, default=None): # x[key] if key in x, else default<br>
> return self.root.get(key, default)<br>
> def has_key(self, key): # True if x has key, else False<br>
> return self.root.has_key(key)<br>
> def items(self): # list of key/val pairs<br>
> return self.root.items()<br>
> def keys(self):<br>
> return self.root.keys()<br>
> def pop(self, key, default=None):<br>
> return self.root.pop()<br>
> def popitem(self): #remove and return an arbitrary key/val pair<br>
> return self.root.popitem()<br>
> def setdefault(self, key, default=None):<br>
> #D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D<br>
> return self.root.setdefault(key, default)<br>
> def values(self):<br>
> return self.root.values()<br>
><br>
> def copy(self): #copy it? dubiously necessary at the moment<br>
> NOT_IMPLEMENTED('copy')<br>
><br>
><br>
> # iteration<br>
> def __iter__(self): # iter(x)<br>
> return self.root.iterkeys()<br>
><br>
> def iteritems(self): #iterator over items, this can be hellaoptimized<br>
> return self.root.iteritems()<br>
><br>
> def itervalues(self):<br>
> return self.root.itervalues()<br>
> def iterkeys(self):<br>
> return self.root.iterkeys()<br>
><br>
> # practical realities of the abstraction<br>
> def garbage_collect(self):<br>
> self.root._p_jar.cacheGC()<br>
> #self.connection.cacheGC()<br>
><br>
> def commit(self):<br>
> return self.__commit_check(force=True)<br>
><br>
> def __commit_check(self, force=False):<br>
> if self.autocommit or force:<br>
> transaction.commit()<br>
><br>
><br>
> On Tue, May 11, 2010 at 3:50 AM, Jim Fulton <<a href="mailto:jim@zope.com">jim@zope.com</a>> wrote:<br>
>><br>
>> On Mon, May 10, 2010 at 8:20 PM, Ryan Noon <<a href="mailto:rmnoon@gmail.com">rmnoon@gmail.com</a>> wrote:<br>
>> > P.S. About the data structures:<br>
>> > wordset is a freshly unpickled python set from my old sqlite oodb<br>
>> > thingy.<br>
>> > The new docsets I'm keeping are 'L' arrays from the stdlib array module.<br>
>> > I'm up for using ZODB's builtin persistent data structures if it makes<br>
>> > a<br>
>> > lot of sense to do so, but it sorta breaks my abstraction a bit and I<br>
>> > feel<br>
>> > like the memory issues I'm having are somewhat independent of the<br>
>> > container<br>
>> > data structures (as I'm having the same issue just with fixed size<br>
>> > strings).<br>
>><br>
>> This is getting tiresome. We can't really advise you because we can't<br>
>> see what data structures you're using and we're wasting too much time<br>
>> guessing. We wouldn't have to guess and grill you if you showed a<br>
>> complete demonstration program, or at least one that showed what the<br>
>> heck your doing.<br>
>><br>
>> The program you've showed so far is so incomplete, perhaps we're<br>
>> missing the obvious.<br>
>><br>
>> In your original program, you never actually store anything in the<br>
>> database. You assign the database root to self.root, but never use<br>
>> self.root. (The variable self is not defined and we're left to assume<br>
>> that this disembodied code is part of a method definition.) In your<br>
>> most recent snippet, you don't show any database access. If you<br>
>> never actually store anything in the database, then nothing will be<br>
>> removed from memory.<br>
>><br>
>> You're inserting data into wordid_to_docset, but you don't show its<br>
>> definition and won't tell us what it is.<br>
>><br>
>> Jim<br>
>><br>
>> --<br>
>> Jim Fulton<br>
><br>
><br>
><br>
> --<br>
> Ryan Noon<br>
> Stanford Computer Science<br>
> BS '09, MS '10<br>
><br>
</div></div><div><div></div><div class="h5">> _______________________________________________<br>
> For more information about ZODB, see the ZODB Wiki:<br>
> <a href="http://www.zope.org/Wikis/ZODB/" target="_blank">http://www.zope.org/Wikis/ZODB/</a><br>
><br>
> ZODB-Dev mailing list - <a href="mailto:ZODB-Dev@zope.org">ZODB-Dev@zope.org</a><br>
> <a href="https://mail.zope.org/mailman/listinfo/zodb-dev" target="_blank">https://mail.zope.org/mailman/listinfo/zodb-dev</a><br>
><br>
><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Ryan Noon<br>Stanford Computer Science<br>BS '09, MS '10<br>
</div>