[ZODB-Dev] what's the latest on zodb/zeo+memcached?

[Marius Gedminas]

On Thu, Jan 17, 2013 at 12:31:52PM -0500, Claudiu Saftoiu wrote:
> I wrote the following code to preload the indices:
> 
>     def preload_index_btree(index_name, index_type, btree):
>         print "((Preloading '%s' %s index btree...))" % (index_name,
> index_type)
>         start = last_print = time.time()
>         for i, item in enumerate(btree.items()):
>             item

That's a no-op: you might as well just write 'pass' here.

If you want to load the btree item into cache, you need to do

              item._p_activate()

>         print "((Preloaded '%s' %s index btree (%d items in %.2fs)))" % (
>             index_name, index_type, i, time.time() - start,
>         )

If you ever get an empty btree, you'll get an UnboundLocalError: 'i' here.

Drop the enumerate() trick and just use len(btree), it's efficient.

>     def preload_catalog(catalog):
>         """Given a catalog, touch every persistent object we can find to
> force
>         them to go into the cache."""
>         start = time.time()
>         num_indices = len(catalog.items())
>         for i, (index_name, index) in enumerate(catalog.items()):
>             print "((Preloading index %2d/%2d '%s'...))" % (i+1,
> num_indices, index_name,)
>             preload_index_btree(index_name, 'fwd', index._fwd_index)
>             preload_index_btree(index_name, 'rev', index._rev_index)
>         print "((Preloaded catalog! Took %.2fs))" % (time.time() - start)
> 
> And I run it on server start as follows (modified for the relevant parts; I
> tried to make the example simple but it ended up needing a lot of parts).
> This runs in a thread:
> 
>     from util import zodb as Z
>     from util import zodb_query as ZQ
>     for i in xrange(3):
>         connwrap = Z.ConnWrapper('index')
>         print "((Preload #%d...))" % (i+1)
>         with connwrap as index_root:
>             ZQ.preload_catalog(index_root.index.catalog)
>         connwrap.close()

Every thread has its own in-memory ZODB object cache, but if you have
configured a persistent ZEO client cache, it should help.

Marius Gedminas
-- 
Never trust a computer you can't repair yourself.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20130118/9a40bfdc/attachment.sig>