[ZODB-Dev] Blob directory structure scalability limits

[Jim Fulton]

On Jun 19, 2008, at 7:38 AM, Christian Theune wrote:

> Hi,
>
> one of our installations hit a scalability limits with the current  
> blob
> directory structure.
>
> The current structure looks like:
>
> \blobs\
>    <oid>\
>        <tid>.blob
>        <tid>.blob
>        ...
>    <oid>\
>        ...
>    ...\
>
> We hit a limit with a database that contains more than 32k blob  
> objects
> because ext3 doesn't allow more than 32k entries in a directory.
>
> We propose to introduce a new mode for the blob storage which breaks  
> the
> directory structure into one level per byte of the oid. This would  
> lead to
> directories 0x00-0xFF nested in 8 levels.

+1

...

> We propose to keep both implementations around and allow to select  
> which one
> to use. We would extend the FileSystemHelper to abstract the two  
> strategies.

-1 to making this selectable.

> We would also provide a migration tool that can convert the old  
> format to the
> new format.

Maybe we can support, but deprecate the old layout and provide a  
conversion scriptt.

Jim

--
Jim Fulton
Zope Corporation