[ZODB-Dev] ZEO and relstporage performance

[Shane Hathaway]

Laurence Rowe wrote:
> Shane's earlier benchmarks show MySQL to be the fastest RelStorage backend:
> http://shane.willowrise.com/archives/relstorage-10-and-measurements/

Yep, despite my efforts to put PostgreSQL on top. :-)  It seems that 
PostgreSQL has more predictable performance and behavior, while MySQL 
wins slightly in raw performance once every surprisingly slow query has 
been optimized.

Thanks to Jim for sending me preliminary results a week ago so I could 
have time to think about it and respond properly. :-)

Still, I think it's fair to point out an issue I discovered in Jim's 
tests.  When Jim expanded my read test to include hot and steamin' 
times, he did not do anything to synchronize the execution of the two 
new test phases.  I can understand that, since the existing structure of 
speedtest.py makes it hard to add synchronization.  However, I 
discovered that the lack of synchronization inflated the "hot" scores by 
a large factor (I've seen up to 4X) under high concurrency, since the 
CPU ends up executing the supposedly concurrent tests at different 
times.  Therefore, the ZEO cache isn't quite as good as Jim's numbers 
suggest.

Even after optimizing the RelStorage/memcached integration, however, my 
own tests show the maximum performance of the ZEO cache is 2 or 3 times 
as fast as the RelStorage/memcached code.  For grins, I closed that gap 
using a ZODB configuration with a fake memcached that stores object 
states in a Python dictionary.

This leads to an interesting question.  Memcached or ZEO cache--which is 
better?  While memcached has a higher minimum performance penalty, it 
also has a lower maximum penalty, since memcached hits never have to 
wait for disk.  Also, memcached can be shared among processes, there is 
a large development community around memcached, and memcached creates 
opportunities for developers to be creative with caching strategies.

So I'm inclined to stick with memcached even though the ZEO cache 
numbers look better.

Shane