[Zope] Size of Data.fs: how big is too big?

[Joachim Werner]

You might also want to serve large files directly via Apache in the long
run, especially if no security restrictions apply for them. On Sun machines
Apache is ways faster at serving large static files (though Zope will
usually handle everything fine if you don't use the server in an intranet.
In a local network, people might well have the high bandwidth to really load
the Zope server with work ...

Joachim

----- Original Message -----
From: "Richard Barrett" <R.Barrett@ftel.co.uk>
To: "Paul Horbal" <horbal@vlsi.enel.ucalgary.ca>; <zope@zope.org>
Sent: Friday, November 30, 2001 12:23 PM
Subject: Re: [Zope] Size of Data.fs: how big is too big?


> At 14:09 29/11/2001 -0700, Paul Horbal wrote:
> >Hi everyone,
> >
> >My Zope site has been growing fairly large lately and I'm beginning to
> >wonder at what point I should consider moving files out of the Zope
filing
> >system and onto a static filesystem.
> >
> >At this point, I haven't noticed any performance issues at
> >all.  Currently, Data.fs is about 300 MB in size.  The site is running on
> >a Sun Netra X1 (400 MHz UltraSparc IIe) with 512 MB of RAM.
> >
> >Should I be worried about adding more large files into Data.fs?
> >
> >thanks,
> >Paul.
>
> Just to relate my decisions on this topic. One of our Zope sites had a
> similar size of Data.fs to yours (and growing fairly rapidly). When I
> looked at the contents it  was clear that a large part of the size was
from
> big blob objects such as PDFs, GIFS and JPEGs, and various Microsoft
> product data files (Word and Powerpoint being favorite).
>
> I wrote an external method which selectively decants the content of
> qualifying objects into ExtFile and ExtImage objects. The big blobs end up
> in the UNIX file system and the stub objects left in Data.fs are much
> smaller; in my case the Data.fs shrank from over 350 Mb to less than 30
Mb.
> I now run the decanting function regularly as well as advising content
> providers to use ExtFile/ExtImage for PDFs and such.
>
> My rationale for this approach was twofold:
>
> 1. I couldn't see any real benefit of inflating Zope's object database
with
> big blobs of fairly static, opaque data. Indeed, my guess was that it was
> more likely to damage the performance of Zope and inflate its process
size,
> although I never tried to prove if this was the case.
>
> 2. Big, fairly static blobs of opaque data adversely affect incremental
> backup performance for the file system containing the Data.fs. A one byte
> change in any object means all the big, unchanged blobs also become
> candidates for being backed up yet again. With the big blobs out in the
> file system, incremental backups go a lot quicker.
>
> This approach won't save in total file space occupied - it wasn't intended
> to - but I figure that it plays to the strengths and away from the
> weaknesses of both the Zope object database and the UNIX file system.
>
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>
>