[ZODB-Dev] ZEO and relstporage performance

[Shane Hathaway]

Jim Fulton wrote:
> These benchmarks make ZEO+FS look pretty good relative to
> MySQLAdapter.  The overall performance assuming even moderate;y
> effective ZEO pr object caches is significantly better for ZEO.

This is an excellent analysis.  It pointed out some rough edges in 
RelStorage performance.  On the RelStorage trunk, I have filled in a lot 
of the performance gaps.  Specifically:

- When writing to the database, RelStorage issued a SQL statement for 
every object stored, causing a network round trip for every object 
written.  I switched it to use multi-row insert statements.

- Allocating a new OID also caused a network round trip.  RelStorage now 
allocates blocks of OIDs.

- I improved the way RelStorage uses memcache.  I reduced the number of 
memcache trips required.  The RelStorage equivalent of the ZEO cache is 
memcache, but memcache wasn't enabled in Jim's tests.  That's ok; 
memcache isn't currently as easy to set up as it should be.

I have also turned the speedtest script into a tool for comparing the 
performance of different ZODB storages with different settings.  Then I 
made a buildout.cfg that installs a copy of mysql, postgresql, memcache, 
and all the necessary adapters so people can easily run tests on their 
own.  I have not released these yet, but I will soon.

Shane