[Zope-dev] Re: any limits on object number?

[Florent Guillaume]

> > thanks to all the responses.  I will test with BTreeFolder2, and
> > subfoldering with first character of id as another subfolder2, so that
> > working with ZMI will also be possible without further scripting or
> > customized views.  i will report the results back to the list.
> 
> just an additional info: I run into this problem recently...
> 
> I've got something around 40k objects to insert into ZODB.
> 
> I started doing some tests with normal Folders, but I discarded
> it completely in favor of BTreeFolders. I got information that
> BTreeFolder was already tested with more than 400k objects...
> and that made me happy! :-)
> 
> But (there's always a 'but'!)...
> 
> Even using BTreeFolder to store all objects I was getting >300s
> delay to show a single object (ok, it's an archetypes-based one,
> containing >50 fields, splitted into 7 schematas, with lots of
> fancy stuff...). So I made a directory hash structure based on
> UID from each object (an AT UID is md5, so we have a hex base).
> 
> Using this hash structure, with 16 divisions, I got a better
> performance (but still far from acceptable): 'only' ~100s to
> show a single object... :-(
> 
> So I decided to create another level for the hash structure:
> now each folder has another 16 folder inside it. This time I
> was getting a delay of ~20s...
> 
> As you're already thinking... it's time for another hash level.
> Now, with 16^3 additional BTreeFolders to split all my objects
> I got an acceptable performance: ~3s...
> 
> The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b,
> but at least the system is usable...
> 
> Resuming what I've learned: don't abuse of BTreeFolder, hash
> your content and live happy forever... ;-)

I very much doubt you solved the problem you think you solved. Access
time to a single object in the hundred of seconds is not a BTreeFolder
problem. BTreeFolder is designed to not be a bottleneck for concurrent
access and large number of objects.

You should have benched (using ZopeProfiler for instance) to find out
where time really is spent. Maybe some of your/AT's code does a stupid
loop on folder.objectIds() or something.

Florent

-- 
Florent Guillaume, Nuxeo (Paris, France)
+33 1 40 33 79 87  http://nuxeo.com  mailto:fg at nuxeo.com