[ZODB-Dev] ZODB open/close profilings

[Jeremy Hylton]

Thanks for doing the leg work, Greg!

>>>>> "GW" == Greg Ward <gward@mems-exchange.org> writes:

  GW> Executive summary:

  GW> * the presence or absence of an index file makes very little
  GW>     difference: 49 sec to open without index, 48 sec to open
  GW>     with

  GW> * it might be worth rewriting the OID packing/unpacking
  GW>     functions in C; opening the database calls U64() 537,404
  GW>     times, accounting for ~12 sec of the ~48 sec total open time

In a ZODB4 branch, I'm experimenting with generous use of the "Q"
struct format code, which does with U64() does.  

  GW> * but the real culprit is FileStorage._sane(), which takes 28
  GW>     sec of the ~48 sec total opening time

I've never thought about the cost of _sane(), so this is a great
clue.  I think part of the problem is that you don't use undo() enough
<wink>.  The code scans the entire data.fs backwards, looking for a
transaction marked with 'u' for undo.  Since you never do an undo, it
reads the whole file backwards after reading it forwards.  Yikes.

Jeremy