[Zope] BTreeFolder for "huge" ZODB (was : populating a "huge" ZODB)

Pierre Godefroy pierre.godefroy@noos.fr
Fri, 21 Jun 2002 12:37:16 +0200

Thank you very much Igor, however I am still a bit afraid because I see a=20
scrolling window for the products contrained by the BTreeFolder on the=20
product homepage (http://www.zope.org/Members/hathawsh/BTreeFolder). In the=
exmple given (4096 items), it could still be OK, but I think having=20
1,000,000 + objects might be a problem. There is an interface problem here=
: the contents of such a folder can only be reasonably displayed in=20
successive batches. (in fact it is also another aspect of the problem of=20
having thousands of objects in an ordinary folder : if you click on the=20
folder to open it, the Management interface will hang)


>On Fre, 2002-06-21 at 11:57, Pierre Godefroy wrote:
>[ huge folder ]
>Have a look at BTreeFolder, the whole structure is stored in a btree, so
>a lookup for an id should be pretty fast...

>Hi everybody,
>How would you populate a "huge" ZODB with hundreds of thousands of objects=
>which have no specific reason to be hierarchically distributed in a=20
>complex tree of folders and sub-folders?
>They could in fact all be in the same folder. It seems to be OK for=20
>hundreds of objects, but afterwards there is an incremental cost : each=20
>time you add a new object, there is a check on the existence of an=20
>identical ID before adding/creating it. When you reach the level of=20
>already existing thousands/tens of thousands/hundreds of thousands objects=
>in the same folder it seems you are in serious trouble...
>I know the question is a bit naive, and that usually you have a data=20
>structure which dictates a naturally "hierarchy" of folders which means=20
>that no folder would contain maybe more than a few hundred objects or=20
>Maybe I should "artificially" design an ID structure which would=20
>correspond to a hierarchy : for instance ojects would be identified by=20
>numbers with 7 digits, each digit corresponding to a lower level in the=20
>hierarchy? It means that an object with ID 2343789 would be placed in=20
>the  "/2/3/4/3/7/8/9" sub-folder.
>What do you think? Any other idea?

Pierre Godefroy
52 rue des Archives
75004 PARIS

T=E9l. : +33 (0)1 42 74 46 05