[ZODB-Dev] Making ZODB / ZEO faster
edahl at zenoss.com
Mon Dec 7 11:08:29 EST 2009
Thanks for all the great feedback. Still processing it but here are
somethings we will try.
RelStorage - in our app context to see if there it helps / hurts. will
report back results. Quick tests show some improvement. We will also
look at tuning out current ZEO setup. Last time I looked there was
only the invalidation queue. I poked around a bit for tuning docs and
didn't see any. Can someone point me to them?
zc.catalogqueue - we are adding fullish indexing to our objects (they
aren't text pages so not truly a "full" index). hopefully moving
indexing out to a separate process will keep the impact of the new
index low and help with our current conflict issues.
Our slow loading object was a persistent with a regular list inside of
the main pickle. Objects that the list pointed to were persistent
which I believe means that hey will load separately. In general we
have tried to make our persistent objects reasonably large to lower
the amount of load round trips. I haven't actually checked its size
yet but it will be interesting to see.
On Dec 4, 2009, at 6:21 PM, Shane Hathaway wrote:
> Jim Fulton wrote:
>> I find this a bit confusing. For the warm numbers, It looks like
>> ZEO didn't
>> utilize a persistent cache, which explains why the ZEO numbers are
>> same for hot and cold. Is that right?
> Yes. It is currently difficult to set up ZEO caches, which I
> consider an issue with this early version of zodbshootout.
> zodbshootout does include a sample test configuration that turns on
> a ZEO cache, but it's not possible to run that configuration with a
> concurrency level greater than 1.
>> What poll interval are you using for relstorage in the tests?
>> Assuming an application gets reasonable cache hit rates, I don't
>> see any
>> meaningful difference between ZEO and relstorage in these numbers.
> You are entitled to your opinion. :-) Personally, I have observed a
> huge improvement for many operations.
>>>> Second, does the test still write and then read roughly the same
>>>> amount of data as before?
>>> That is a command line option. The chart on the web page shows
>>> reading and
>>> writing 1000 small persistent objects per transaction,
>> Which is why I consider this benchmark largely moot. The database is
>> small enough
>> to fit in the servers disk cache. Even the slowest access times are
>> on the order of .5
>> milliseconds. Disk accesses are typically measured in 10s of
>> milliseconds. With magnetic
>> disks, for databases substantially larger than the server's ram, the
>> network component
>> of loading objects will be noise compared to the disk access.
> That's why I think solid state disk is already a major win,
> economically, for large ZODB setups. The FusionIO cards in
> particular are likely to be at least as reliable as any disk. It's
> time to change the way we think about seek time.
More information about the ZODB-Dev