[ZODB-Dev] Large BTrees

[Gfeller Martin]

Tim, thank you very much for your hints.

Most of your objects are persistent on their own, and after your
explanations,
I will convert those that escaped. 

>> memory profiling is outright hard in Python
>As compared to what -- Perl <wink>?  You're so many layers removed from
the
>hardware that I suspect this is much more a function of general
language
>level than of Python specifically.  Even a Python dict in isolation is
a
>non-trivial data structure.

I used APL in the nineties, certainly a high level, interpreted language
with
non-trivial data structures (and as readable as Perl ;-)). 

Support for memory profiling was good (space consumed by each object 
"on its own", space of each accessible objects), but memory was much
more of 
a constraint at that time, so support *had* to be good.

Best regards,
Martin
www.quantax.com

-----Original Message-----
From: Tim Peters [mailto:tim at zope.com] 
Sent: Tuesday, 16 Dec 2003 01:19
To: Gfeller Martin
Cc: zodb-dev at zope.org; Christian Robottom Reis
Subject: RE: [ZODB-Dev] Large BTrees (WAS: IIBTree.multiunion and
listcomprehensions)


[Gfeller Martin]
> I should have said that processing a 100000 deals BTree takes quite
> a while in Quantax (30 minutes), and this is acceptable, as we also do
> some quite involved calculations.
>
> My major concern and tuning is on memory, not speed (2 GB limit), and
> I spent considerable effort to call cache collections at the right
> points during the transaction (and ensuring that cached objects
> actually are released, without making references to them to disappear
> "too fast" - hence my usage of _v_attributes to store "back pointers"
to
> weak refs - which I mentioned a couple of days ago).

Without knowing anything about the data access and data modification
patterns involved in the 30-minute computation, of course it's hard to
suggest anything helpful.

> I find memory profiling is outright hard in Python

As compared to what -- Perl <wink>?  You're so many layers removed from
the
hardware that I suspect this is much more a function of general language
level than of Python specifically.  Even a Python dict in isolation is a
non-trivial data structure.

> - any hints appreciated.

Recent Python releases have a file, Misc/SpecialBuilds.txt, that
describes
some "non-standard" ways to build Python to get extra low-level info.
Since
Python doesn't give you direct access to malloc(), "memory" is usually
less
interesting to look at than, e.g., per-type object highwater marks (see
the
COUNT_ALLOCS special build).  BTW, this is all more useful the more
recent
the Python, and the more new-style classes are used.  PYTHONMALLOCSTATS
can
be used in a debug build to get a detailed picture of bottom-level raw
memory used for "small objects", but wrestling at that level is
frustrating
because you're not *programming* at that level.

> Wrt time, I noticed we spend actually quite some time of
> Connection.setstate in actual file reading (or in Windows overhead);
> silly things as turning off the Virus Checker on the DB file brought
> me 30% speedup...

My favorite is screen savers:  when you find that overnight runs take
longer, or act flakier, than runs you stare at during the day, throwing
away
a screen saver can work wonders.

> I'm using OOBtrees with string-tuples as keys (~50 bytes) and objects
> of about 15 KB as values.

Are the objects persistent on their own?

All the data in BTrees lives in leaf-level buckets, and an OOBTree holds
a
maximum of 30 key+value pairs in each bucket.  Depending on how the
BTree
was built up, it may average closer to 15 pairs per bucket.  If these
objects aren't persistent on their own, then their full states have to
be
saved with the bucket, and 15 values/bucket * 15 KB/value is about a
quarter
MB of pickle.  Then reading up one bucket and modifying one value in it
adds
up to a half MB of I/O throughput, and probably at least a half MB of
RAM
too (the .25 MB pickle string will be in RAM as long as it takes to
create
the probably-greater-than-.25 MB bucket object).

If the objects are persistent on their own, good, then the bucket
pickles
can be much smaller, and loading or changing one value doesn't have to
unpickle the full state for 15 to 30 values.

> I measured that venerable PersistentMapping sometimes is much faster,
> especially on first sequential reads (after the cache is cleared), but
> consumes more memory.

Without knowing anything about the data access and data modification
patterns, anything is believable <wink>.

> So, no silver bullet, but you didn't expect that :-)

Ditto.