[ZODB-Dev] performance of chunked data over ZEO

[Tim Peters]

[Andreas Jung]
> Inside an application that stored large amounts of data in the ZODB as
> linked structure (very similar to the Pdata magic in OFS.Image) I
> discovered that the chunk size has a significant influcence on the
> performance.  Using a chunk size of 16k (which is also the default in
> OFS.Image)

Looks like 64KB is the actual chunk size in OFS.Image.

> I got the expected transfer rate of about 2-3MB/second.

What does "expected" mean?  That is, why would you expect any particular
rate?  I think we've been very careful not to document anything about what's
expected here <wink>.

> Increasing the chunk size to 64K caused a performance loss to about
> 100-150 KB/seconds. Any ideas why increasing the chunk size causes such a
> dramatic performance loss?

Not really; best guesses are that one case is exploiting a cache the other
case isn't, or that it will resolve to some inscrutable detail of how socket
TCP is implemented on your (unidentified) platform.  For example, whenever
TCP figures vary wildly for no apparent reason, speculation about Nagle's
algorithm is traditional.  Jeremy tried to cut that off by having ZEO do its
own layer of packet chunking, but who knows.

If you can create a small, self-contained test case, and post it here, at
least people could see how it behaves on a variety of platforms.  Try a
variety of chunk sizes too, as Dieter did.  Note that if you can't reproduce
the symptom (as Dieter wasn't able to), then the test code still isn't
modeling something crucial about what your real app is doing, and then
you're likely the only one who could guess what that might be.  OTOH, if a
simple test case does reproduce your symptoms on your box, but not on other
boxes, then that's a good clue.