[ZODB-Dev] FileStorage size/density survey

[Casey Duncan]

You might be able to approach this in only a somewhat more sophisticated =
way=20
with good results. Break the full index array up into N files containing =
a=20
"page" of the index (basically just a fragment of the direct lookup array=
).=20
Write an index class that fronts for these files, creating, reading and=20
writing them as needed. Only index pages that have data in them would nee=
d to=20
exist on disk (or in memory).

It could have a pool of index page file descriptors that are kept open at=
 one=20
time. Both the pool size and the page file size could be configurable. Th=
en=20
memory and time could be traded at will to tune it. And it would still en=
d up=20
being reasonably simple to implement I think.

-Casey

On Tuesday 17 December 2002 10:46 am, Greg Ward wrote:
> Hi all -- I'm toying with the idea of implementing a new index type for
> FileStorage as an alternative to both the in-memory Python dict
> currently used, and the on-disk BTree planned for ZODB 3.2.  Right now
> I'm drawing pictures and estimating memory use -- nice thing about
> implementing data structures in C, you can make reasonable guesses abou=
t
> memory use.  ;-)
>=20
> AFAICT, the only variables that affect the size of the index are the
> number of "slots" in a FileStorage (ie. highest OID used +1) and the
> actual number of objects.  (Actually, the number of "slots" only matter=
s
> for a dead-simple direct-lookup array, which I'm only considering as a
> hypothetical fast, simple, and memory-inefficient approach.)
>=20
> I'd like to find out what these numbers are for a variety of real world
> ZODB databases.  So could you please run the attached script on any ZOD=
B
> .fs files you happen to have lying around and mail me the results?  I'l=
l
> summarize to the list if anyone else is interested.
>=20
> Thanks --
>=20
>         Greg
> --=20
> Greg Ward - software developer                gward@mems-exchange.org
> MEMS Exchange                            http://www.mems-exchange.org
>=20