[ZODB-Dev] standalone ZODB for large Numeric arrays
Mike Mueller
mmueller@dgfz.de
Mon, 03 Mar 2003 10:14:15 +0100
At 22:09 28.02.03 +0100, you wrote:
[Kaspar]
>What is the main reason you want this?
>Would using netCDF from inside ZODB really be faster than what ZODB does now?
I haven't really measured yet. But netCDF is designed for large arrays and
fast access.
If ZODB.FileStorage files are big (> 1GB) it takes quite a while to open
them (30 seconds, 2.2 GHZ Pentium 4, 1 GB RAMBUS). A netCDF file opens just
like a ordinary file, at least in terms of speed. Also, since ZODB makes
undo possible, files get large rather quickly. With netCDF I can use large
file support (> 2GB) or just distribute the data over several files using
ZODB to keep track in which netCDF file things are stored.
>How do you want to use the netCDF functionality? From inside Python or from
>other software?
Right now I use netCDF from FORTRAN also. Actually, I put arrays in netCDF
on a RAM disk from FORTAN and read them back in Python and vice versa. This
is something like a compiler independent data exchange. The overhead is an
additional copy plus some netCDF operations.
>BTW, I just started using ZODB today, I'm also using it for numerical
>calculations and I really think it kicks ass!
I agree. It's very convenient. For instance, I use mx.DateTime objects as
keys to store data at different dates. No need to convert the date into
seconds since some starting point or something like this.
Mike