[ZODB-Dev] Making ZODB / ZEO faster

[Erik Dahl]

Right I have looked at the nosql stuff a good bit.  MongoDB is  
interesting but no transactions and their sharding is alpha code.   
ChouchDB has transactions but no sharding at all.  HBase has what I  
want but I can't get my brain around how to use the wacky schema.

Unfortunately, this isn't a new project so if we could get more out of  
what we have that would be great but clearly swapping out the back end  
may become necessary.

I haven't dove into the relstorage yet but have heard it will perform  
better.  Not sure I understand why though.  Isn't it just putting  
pickles into a single table with an index on the oid? (or oid / serial).

-EAD



On Dec 4, 2009, at 10:15 AM, Andreas Jung wrote:

> The project - at least enterprise-level projects - requires are  
> careful
> choice of the tools and backends. The ZODB is the golden bullet for  
> all
> and everything. Depending on the data model and the project needs you
> have to
> look at relational database or NOSQL databases as alternatives.
>
> And as you wrote: the ZODB-based application perform bad in heavy- 
> write
> scenarios...you may raise the limits by using Relstorage but perhaps
> other backends might be the better choice - something one must  
> consider
> when starting a new project.
>
> -aj
>
> Am 04.12.09 15:41, schrieb Erik Dahl:
>> Guys,
>>
>> We have a product written in python using ZODB/ZEO and I would like  
>> to
>> improve the speed of database in general.  Things that I have seen
>> that I would like to improve some I understand and some not.
>>
>> 1. Loading of largish (but not too large object had a list with  
>> around
>> 20K references in it) pickles can be very slow.  Ok what size pickle?
>> not 100% sure is there a way to get the pickle size of a zodb
>> persistent object?  I lamely tried to pickle one of our persistent
>> objects and of course it blew up with max recursion because it went
>> beyond the normal bounds of a zodb persistent pickle.  There must  
>> be a
>> way to do this though right?
>>
>> 2. Writing lots of objects.  I know that zodb wasn't written for this
>> type of use case but we have backed into it.  We can have many (~30  
>> is
>> that a lot?) zodb clients and as a result large numbers of cache
>> invalidations can be sent when a write occurs.  Could invalidation
>> performance / cache refresh be an issue?
>>
>> 3. DB hot spots.  Of course we see conflict errors when there are  
>> lots
>> of writes to the db from different clients that touch the same
>> object.  We haven't done a bunch of optimization work here but I'm
>> thinking of moving all indexing out to a separate client/process that
>> reads off a queue to find objects to index.  I'm guessing the indexes
>> are a hotspot (haven't tested this out much though I guess b-tree's
>> buckets should alleviate this problem some).  (is there a persistent
>> queue around?)
>>
>> Anyway these are some things that come to mind when I think of
>> performance issues.  I have the thought that many could be made  
>> better
>> with faster ZEO I/O.  Does this seem like a good assumption?  If so
>> what could we do to make ZEO faster?
>>
>> Questions:
>>
>> * We use a filestorage are there faster ones?  Can this be a  
>> bottleneck?
>> * Is the ZEO protocol inefficient?
>> * Is the ZEO server just plain slow?
>>
>> Thoughts I have that may have no impact.
>>
>> * rewrite ZEO or parts of it in C
>> * write a C based storage
>>
>> Others?
>>
>> -EAD
>>
>>
>>
>> _______________________________________________
>> For more information about ZODB, see the ZODB Wiki:
>> http://www.zope.org/Wikis/ZODB/
>>
>> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
>> https://mail.zope.org/mailman/listinfo/zodb-dev
>>
>
>
> -- 
> ZOPYX Ltd. & Co KG          \  zopyx group
> Charlottenstr. 37/1          \  The full-service network for your
> D-72070 Tübingen              \  Python, Zope and Plone projects
> www.zopyx.com, info at zopyx.com  \  www.zopyxgroup.com
> ------------------------------------------------------------------------
> E-Publishing, Python, Zope & Plone development, Consulting
>
>
> <lists.vcf>