[ZODB-Dev] Large BTrees (WAS: IIBTree.multiunion and listcomprehensions)

[Tim Peters]

[Gfeller Martin]
> I should have said that processing a 100000 deals BTree takes quite
> a while in Quantax (30 minutes), and this is acceptable, as we also do
> some quite involved calculations.
>
> My major concern and tuning is on memory, not speed (2 GB limit), and
> I spent considerable effort to call cache collections at the right
> points during the transaction (and ensuring that cached objects
> actually are released, without making references to them to disappear
> "too fast" - hence my usage of _v_attributes to store "back pointers" to
> weak refs - which I mentioned a couple of days ago).

Without knowing anything about the data access and data modification
patterns involved in the 30-minute computation, of course it's hard to
suggest anything helpful.

> I find memory profiling is outright hard in Python

As compared to what -- Perl <wink>?  You're so many layers removed from the
hardware that I suspect this is much more a function of general language
level than of Python specifically.  Even a Python dict in isolation is a
non-trivial data structure.

> - any hints appreciated.

Recent Python releases have a file, Misc/SpecialBuilds.txt, that describes
some "non-standard" ways to build Python to get extra low-level info.  Since
Python doesn't give you direct access to malloc(), "memory" is usually less
interesting to look at than, e.g., per-type object highwater marks (see the
COUNT_ALLOCS special build).  BTW, this is all more useful the more recent
the Python, and the more new-style classes are used.  PYTHONMALLOCSTATS can
be used in a debug build to get a detailed picture of bottom-level raw
memory used for "small objects", but wrestling at that level is frustrating
because you're not *programming* at that level.

> Wrt time, I noticed we spend actually quite some time of
> Connection.setstate in actual file reading (or in Windows overhead);
> silly things as turning off the Virus Checker on the DB file brought
> me 30% speedup...

My favorite is screen savers:  when you find that overnight runs take
longer, or act flakier, than runs you stare at during the day, throwing away
a screen saver can work wonders.

> I'm using OOBtrees with string-tuples as keys (~50 bytes) and objects
> of about 15 KB as values.

Are the objects persistent on their own?

All the data in BTrees lives in leaf-level buckets, and an OOBTree holds a
maximum of 30 key+value pairs in each bucket.  Depending on how the BTree
was built up, it may average closer to 15 pairs per bucket.  If these
objects aren't persistent on their own, then their full states have to be
saved with the bucket, and 15 values/bucket * 15 KB/value is about a quarter
MB of pickle.  Then reading up one bucket and modifying one value in it adds
up to a half MB of I/O throughput, and probably at least a half MB of RAM
too (the .25 MB pickle string will be in RAM as long as it takes to create
the probably-greater-than-.25 MB bucket object).

If the objects are persistent on their own, good, then the bucket pickles
can be much smaller, and loading or changing one value doesn't have to
unpickle the full state for 15 to 30 values.

> I measured that venerable PersistentMapping sometimes is much faster,
> especially on first sequential reads (after the cache is cleared), but
> consumes more memory.

Without knowing anything about the data access and data modification
patterns, anything is believable <wink>.

> So, no silver bullet, but you didn't expect that :-)

Ditto.