[ZODB-Dev] what's the latest on zodb/zeo+memcached?

[Claudiu Saftoiu]

On Fri, Jan 18, 2013 at 9:02 AM, Marius Gedminas <marius at gedmin.as> wrote:

> On Thu, Jan 17, 2013 at 12:31:52PM -0500, Claudiu Saftoiu wrote:
> > I wrote the following code to preload the indices:
> >
> >     def preload_index_btree(index_name, index_type, btree):
> >         print "((Preloading '%s' %s index btree...))" % (index_name,
> > index_type)
> >         start = last_print = time.time()
> >         for i, item in enumerate(btree.items()):
> >             item
>
> That's a no-op: you might as well just write 'pass' here.
>

True, I wanted to do something with 'item' but didn't know what.



> >         print "((Preloaded '%s' %s index btree (%d items in %.2fs)))" % (
> >             index_name, index_type, i, time.time() - start,
> >         )
>
> If you ever get an empty btree, you'll get an UnboundLocalError: 'i' here.
>
> Drop the enumerate() trick and just use len(btree), it's efficient.
>

Thanks for catching that. `len` still takes a while on a large btree though
if it isn't in memory:

    In [7]: start = time.time(); len(bt); end = time.time()
    Out[7]: 350169
    In [8]: end - start
    Out[8]: 32.397267818450928

It actually seems to require loading the entire tree, because after running
`len`, subsequent operations (like iterating through the entire tree) start
happening instantly. However, since I just iterated through the entire
tree, it will definitely be fast at that point.


> If you want to load the btree item into cache, you need to do
>
>               item._p_activate()
>

That's not going to work, since `item` is a tuple. I don't want to load the
item itself into the cache, I just want the btree to be in the cache. I
figured iterating through the entire tree would force it to be loaded, but
is that not the case? If not then what should I call `_p_activate()` on? I
assume calling it on the tree itself won't cause all its internals to be
loaded. I'm not familiar with the internals of the BTree, however. Would
this be a better solution?

    def preload_index_btree(index_name, index_type, btree):
        print "((Preloading '%s' %s index btree...))" % (index_name,
index_type)
        start = time.time()
        num_buckets = 0
        bucket = btree._firstbucket
        while bucket:
            bucket._p_activate()
            num_buckets += 1
            bucket = bucket._next
        print "((Preloaded '%s' %s index btree (%d/%d buckets items in
%.2fs)))" % (
            index_name, index_type, len(btree), num_buckets, time.time() -
start,
        )

>     def preload_catalog(catalog):
> >         """Given a catalog, touch every persistent object we can find to
> > force
> >         them to go into the cache."""
> >         start = time.time()
> >         num_indices = len(catalog.items())
> >         for i, (index_name, index) in enumerate(catalog.items()):
> >             print "((Preloading index %2d/%2d '%s'...))" % (i+1,
> > num_indices, index_name,)
> >             preload_index_btree(index_name, 'fwd', index._fwd_index)
> >             preload_index_btree(index_name, 'rev', index._rev_index)
> >         print "((Preloaded catalog! Took %.2fs))" % (time.time() - start)
> >
> > And I run it on server start as follows (modified for the relevant
> parts; I
> > tried to make the example simple but it ended up needing a lot of parts).
> > This runs in a thread:
> >
> >     from util import zodb as Z
> >     from util import zodb_query as ZQ
> >     for i in xrange(3):
> >         connwrap = Z.ConnWrapper('index')
> >         print "((Preload #%d...))" % (i+1)
> >         with connwrap as index_root:
> >             ZQ.preload_catalog(index_root.index.catalog)
> >         connwrap.close()
>
> Every thread has its own in-memory ZODB object cache, but if you have
> configured a persistent ZEO client cache, it should help.
>

Gotcha. Thanks for the help!
- Claudiu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20130118/6d1aa3cf/attachment.html>