[Zope3-dev] DoS problem in zope.sendmail.maildir

[Marius Gedminas]

Hi,

On Wed, Jun 06, 2007 at 06:04:50PM +0200, Sascha Ottolski wrote:
> I discovered a problem with the maildir implementation, that is 
> triggered when trying to use the QueuedDelivery with many (read: more 
> than current filedescriptor limit of the zope process) mails at once:
...
> OSError: [Errno 24] Too many open files: '/home/so/sandbox/var/zope/lib/python/lalisio/app/site/../../../../var/zope/var/mqueue/new'

Oh, wow.  I'd forgotted that this causes the data manager to keep an
open file descriptor for every email.  Ouch.

Suggestion: introduce IMaildirMessageWriter.close() and call it in
QueuedMailDelivery.createDataManager.

> I need to kill -9 the server after that, SIGTERM seems to be ignored.

Ouch.

> The reason for that to happen is clear to me, however, I'm not sure how
> a fix could look like.

Did you create an bug in launchpad?  I could attach a patch that I think
would fix the problem, but I don't currently have the time and energy to
test myself.

> One source of the problem is
> 
> 
> try:
>     fd = os.open(filename, os.O_CREAT|os.O_EXCL|os.O_WRONLY, 0600)
> except OSError:
>     # File exists
> 
> 
> OSError is obviously to broad to catch as a test if the unique filename 
> could be created. May be use os.path.exists(filename) instead? But I'm 
> not sure if this is equivalent in a platform independant way.

That solution would have a race condition.  Better

  try:
      fd = os.open(filename, os.O_CREAT|os.O_EXCL|os.O_WRONLY, 0600)
  except OSError, e:
      import errno # actually put this at the very top
      if e.errno != errno.EEXIST:
          raise
      # File exists

but would it work on Windows?  Could anyone test that by running this
little program on Windows:

import os, errno
file('tempfile', 'w').close()
try:
    fd = os.open('tempfile', os.O_CREAT|os.O_EXCL|os.O_WRONLY, 0600)
except OSError, e:
    print e.errno == errno.EEXIST # prints True on Linux

> The real source of the problem lies in the MaildirMessageWriter: The 
> passed-in filedescriptor remains open until commit() is called. No 
> problem for sending some mails only, but when sending lots of 
> messages in one transaction, the filedescriptor limit kicks in.

It's scary to see the problems in zope.sendmail, given that I'm
at least partially responsible for all of them.

> A dirty solution might be to change the write() and writeline() to open 
> and close the file on every call. But there's probably a saner way. 

Yes.

> BTW, I'm wondering if it is really safe to assume that 
> os.rename(self._filename, self._new_filename) in the commit() message 
> never fails?

Nothing is really safe.  Renaming a file is much safer than creating a
file.

> Although I believe that it's almost impossible that it 
> really could :-)

Sudden filesystem corruption, the kernel remounts it as read-only, and
you cannot rename anything.  But then you have bigger problems than an
exception in the second phase of a commit.

Marius Gedminas
-- 
For Sale: Parachute.  Only used once, never opened, small stain.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope3-dev/attachments/20070607/2944ff09/attachment.bin