[ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

[Andreas Jung]

--On 29. Mai 2005 11:29:06 +0200 Christian Theune <ct at gocept.com> wrote:

> Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes:
>> Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's
>> OFS.Image.Image class (and also Zope3's implementation) is using a so
>> called possible large data class (Pdata) that is a subclass of
>> Persistent.
>>
>> Pdata is using a simple and genious approach to minimize the memory
>> usage when storing large binary data in ZODB. The data is read from a
>> [...]
>
> Actually Pdata has some drawbacks. When the blobsupport branch gets
> declared stable (I think it's not gonna happen in 3.4, but nobody told
> me otherwise) we'll have really good blob support without this black
> magic.
>

The Pdata approach in general is not bad. I have implemented a CVS-like file
repository lately where we store binary content using a pdata like 
structure.
Our largest files are around (100MB) and the performance and efficiency is 
not bad
although it could be better. The bottleneck is either the ZEO communication 
or just the network.
I reach about 3.5 MB/second while reading such a large file from the ZEO 
server.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20050529/95e3d5db/attachment.bin