Metadata in zope / binary data on FS, was Re: [ZODB-Dev] __del__ with Persistent objects

[Paul Winkler]

On Fri, Jul 18, 2003 at 10:45:41PM +1000, Stuart Bishop wrote:
> ... Using FileStorage for the metadata and
> indexes, and storing blobs externally, seems sane. I don't see a point
> in dealing with a 20GB+ Data.fs if I don't have to :-) Added advantages
> are being able to trivially recover the documents from backup tapes, and
> to archive files no longer referenced in the database to permanent
> storage (By moving them to an archive directory when they are no longer
> referenced in the ZODB).

The problem is not just storage bloat, I've found that Zope 
performance simply sucks when serving large files. A couple of
simultaneous requests for a 40 MB file can make zope crawl :-(
So I'm needing to implement something similar, and wondering...

1) is there any existing zope product that does this job? metadata in
zope, binary files on disk served by apache or whatever?

2) has anybody used APE on a production site to handle filesystem 
storage of large files? If so, I think this would simplify the job -
it might suffice to derive from the normal zope File class and
simply override index_html, so that it redirects to a URL at which
Apache serves the same filesystem content.

My sites are read frequently and written relatively rarely, so I'm going
to assume that uploads are not a performance problem.

The biggest problem I see: security. 
First of all, both apache and zope need to read the binary files, but I 
don't want apache to be able to read just anything in the zope tree.
Worse, some of my binary files need to be password-protected :-\
OK, mod_ldap and LdapUserFolder should take care of centralized
authentication, but how the heck are my content managers going to control
access to the apache-served stuff?  Maybe the zope product could
generate an .htaccess file based on security settings? hell, I dunno.

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's SECRETARY EYE!
(random hero from isometric.spaceninja.com)