[Ape] APE issue
shane at hathawaymix.org
Thu Oct 21 04:42:29 EDT 2004
On Tuesday 12 October 2004 02:21 am, Andrew Veitch wrote:
> I've been trying to migrate a 5GB ZODB onto APE/filesystem. It's worked
> fine on the whole but I've had a few issues:
> First, a dozen or so of files in the ZODB are quite large, 100MB-200MB. I
> know that neither ZODB nor APE are really designed for files of this size
> but it works fine under ZODB although it is a little slow sometimes.
> However under APE we frequently get 'out of memory' errors - both when
> creating and viewing the files. The server has 4GB of RAM plus swap space
> and I was the only user at the time so I am surprised that a 100MB file
> causes this error. Unfortunately there is no traceback in the log - I
> guess that running out of memory stops the logging.
I imagine there was some limit imposed on the process that caused it to fail
long before reaching 4 GB. Perhaps the kernel can not create a single
process of such size. Regardless, Ape needs to handle large files better.
It actually has the potential to do an excellent job with large files, I
think, but it currently punts.
Right now, Ape loads and saves all files as a big chunk. We could make File
objects read and write the filesystem directly when Ape is running. The one
complication is preserving transactions, but I hope that's not too
The first part of the code that needs to change somehow is the FilePData class
in apelib.zope2.ofsserial. It currently converts the "data" attribute
between a chain of PData objects (see OFS.Image) and a simple string.
FilePData could sneakily replace the "data" attribute with something of its
own making--something that reads the data directly from a file. Then it
could pass the filename instead of the file data to and from the gateway. Do
you see what I mean?
> Secondly, I've occasionally had 'maximum recursion depth exceeded'
> messages when pasting content. This is when pasting 4 or 5 files.
> Traceback appended. This looks to me like a problem with Archetypes and
> the Pickler rather than APE per se. But again, it has never happened with
> the ZODB. Increasing Python's maximum recursion did resolve the issue.
This is probably the correct resolution. Ape uses deeper pickles than
More information about the Ape