[Zope] FYI: large files & linux.

[Ty Sarna]

In article <87n1oazuao.fsf@erwin.complete.org>,
John Goerzen  <jgoerzen@complete.org> wrote:
> It supports it to no greater or less extent than Linux does.  The

Untrue.

> standard library defaults to using an int or a long for *seek calls.
> The difference is semantic only, as they're both 32 bits on both
> systems anyway.  Even if you assume unsigned (fseek is signed), that's
> still 4gb, max.

The kernel internals of Linux use a 32 bit off_t, hence discussion of
large file support in the next kernel release. 

> I submit, therefore, that you are not seeing a Linux problem, but an
> x86 problem :-)

I'm not seeing an x86 problem at all:

lukasiewicz$ uname -sp
NetBSD i386
lukasiewicz$ ls -l huge
-rw-r--r--  1 tsarna  users  17592186044415 Mar  7 12:41 huge

Granted, that's a sparse file (I can't afford that much disk space :^),
but it demonstrates that file offsets are not a problem.  As mentioned,
NetBSD's off_t has been a 64 bit type for about 6 years now, even on
i386 (and evey other platform).  In fact, off_t was made 64 bit in 1994,
and NetBSD/alpha didn't exist until 1995, so it was done before there
were any native 64 bit platforms supported.  :-)

However, it turns out that for Zope use, at least with FileStorage,
there is a problem, which turns out to be even an x86 issue but a silly
ANSI C spec issue.  Unfortunaly Python large file support is broken on
(32-bit CPU) NetBSD as a result -- I had to make that sparse file with a
little C program.  Now, you could have a more than 4G BerkeleyStorage
Zope system, since the file handling isn't done at the Python level in
that case, but FileStorage won't work until I can work around the Python
problem (or until NetBSD 1.5, which will fix it in a different way). 

But at least it's the kind of thing that could be hacked around in an
application by one guy in a day, rather than taking drastic kernel
and libc changes :-)