[ZODB-Dev] Blob directory structure scalability limits

[Christian Theune]

Hi,

one of our installations hit a scalability limits with the current blob
directory structure.

The current structure looks like:

\blobs\
    <oid>\
        <tid>.blob
        <tid>.blob
        ...
    <oid>\
        ...
    ...\

We hit a limit with a database that contains more than 32k blob objects
because ext3 doesn't allow more than 32k entries in a directory.

We propose to introduce a new mode for the blob storage which breaks the
directory structure into one level per byte of the oid. This would lead to
directories 0x00-0xFF nested in 8 levels. 

The last directory denotes the blob itself and looks like the current
directory: a list of blob files named by the tids they were committed for.

We propose to keep both implementations around and allow to select which one
to use. We would extend the FileSystemHelper to abstract the two strategies.

We would also provide a migration tool that can convert the old format to the
new format.

Comments?

Christian


-- 
Christian Theune · ct at gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1
Zope and Plone consulting and development