[Zope3-dev] DoS problem in zope.sendmail.maildir

[Sascha Ottolski]

Hi,

I discovered a problem with the maildir implementation, that is 
triggered when trying to use the QueuedDelivery with many (read: more 
than current filedescriptor limit of the zope process) mails at once:


Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
    self.run()
  File "/home/so/sandbox/app/zope3/lib/python/zope/sendmail/delivery.py", line 193, in run
  File "/home/so/sandbox/app/Zope-3.3.0/lib/python/zope/sendmail/maildir.py", line 69, in __iter__
OSError: [Errno 24] Too many open files: '/home/so/sandbox/var/zope/lib/python/lalisio/app/site/../../../../var/zope/var/mqueue/new'


Which then seems to make the whole server blocking; trying to open a 
page on the the server results in:


Traceback (most recent call last):
  File "/home/so/sandbox/app/zope3/lib/python/twisted/python/log.py", line 43, in callWithContext

  File "/home/so/sandbox/app/zope3/lib/python/twisted/python/context.py", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/so/sandbox/app/zope3/lib/python/twisted/python/context.py", line 37, in callWithContext
    return func(*args,**kw)
  File "/home/so/sandbox/app/zope3/lib/python/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite

--- <exception caught here> ---
  File "/home/so/sandbox/app/zope3/lib/python/twisted/internet/tcp.py", line 753, in doRead

  File "/usr/lib/python2.4/socket.py", line 161, in accept

socket.error: (24, 'Too many open files')


I need to kill -9 the server after that, SIGTERM seems to be ignored.

The reason for that to happen is clear to me, however, I'm not sure how
a fix could look like.

One source of the problem is


try:
    fd = os.open(filename, os.O_CREAT|os.O_EXCL|os.O_WRONLY, 0600)
except OSError:
    # File exists


OSError is obviously to broad to catch as a test if the unique filename 
could be created. May be use os.path.exists(filename) instead? But I'm 
not sure if this is equivalent in a platform independant way.

The real source of the problem lies in the MaildirMessageWriter: The 
passed-in filedescriptor remains open until commit() is called. No 
problem for sending some mails only, but when sending lots of 
messages in one transaction, the filedescriptor limit kicks in.

A dirty solution might be to change the write() and writeline() to open 
and close the file on every call. But there's probably a saner way. 

BTW, I'm wondering if it is really safe to assume that 
os.rename(self._filename, self._new_filename) in the commit() message 
never fails? Although I believe that it's almost impossible that it 
really could :-)


Thanks, 

Sascha

-- 
Lalisio - connecting knowledge worldwide

Lalisio GmbH
Puschkinstraße 1
99084 Erfurt
fon +49 (0)361 541 43 80
fax +49 (0)361 541 43 79
kontakt at lalisio.com
www.lalisio.com

Sitz der Gesellschaft: Erfurt
Amtsgericht Jena, HRB 113943
Geschäftsführerin: Ute Rother
Ust-Id.: DE813237441