[Zope3-dev] Re: Big Problem with FTP in Zope 3.3.0, seems to be the result of using `v = self.get(name, self); if v is self...` in a land where Proxy objects can kill identity comparison

[Jeff Shell]

I have to follow this up. While I thought I was doing a good job
reporting and testing the problem, I didn't do a hard enough
control-group study. So, I started to investigate further:

Even though I saw absolutely nothing about our ReadDirectory subclass
that would cause the strange behavior, I decided to double check the
behavior against a fresh install of Zope 3.3.0, with only my
pdb.set_trace statement in place (this time I placed it in
``zope.app.filerepresentation.ReadDirectory.__getitem__`` so I could
watch traversal).

Trying to chdir to '/customer/bogus' in FTP yielded the desired result::

    ftp> cd customer/bogus
    550 /customer/bogus: No such file or directory.

After seeing that, I stepped through the debugger. Sure enough, the
``if v is self`` failed in ``ReadDirectory.__getitem__``, causing the
situation I explained before:

Basically, in the FTP situation (or any situation wherein proxies are
being made), the following scenario happens. The _dir is used in place
of 'self' to show that it's what ultimately gets returned to the
publisher::

1. publishTraverse('bogus') from customer
2. _dir = IReadDirectory(customer)
3. _dir['bogus'] is called

    a. v = _dir.get('bogus', _dir) [returns _dir]
    b. ``if v is _dir: raise KeyError('bogus')`` fails, as v and _dir
are the same proxied objects, but have diferent identities at the
Python level.
    c. Instead of raising a KeyError on the bogus (ha!) name, ``_dir``
is returned.

So this phantom ReadDirectory object gets sent out to the publisher.
The same situation is happening on a clean Zope 3.3.0 instance and on
the instance on which we found the problem (which provides a
subclassed ReadDirectory adapter for our container objects). If Zope's
ReadDirectory and Bottlerocket's ContentContainerReadDirectory are
being pushed up to the publisher as phantoms in each case, why does
the default setup return the proper FTP "No such file" error where the
Bottlerocket setup allows one to enter this non-existing directory?

I traced the behavior down to
``zope.app.publication.ftp.FTPPublication`` and the *callObject*
method. `callObject` essentially is doing this (staying with the
'_dir' object mentioned above)::

    _dir = ob
    view = queryMultiAdapter((ob, ftprequest), name='ls', default=self)

Ignoring the ``default=self`` here, `queryMultiAdapter` **fails on the
default setup, but returns FTPView in our custom setup!** Again - we
do nothing special in our ZCML registration, and our custom setup
subclasses from the default ReadDirectory, what's going on? I decided
to look at the list of provided interfaces at the moment of
``FTPPublication.callObject``

The behavior of a *clean setup*, doing ``chdir ... bogus``::

    (Pdb) pp list(providedBy(ob).__sro__)
    [<implementedBy zope.app.folder.filerepresentation.ReadDirectory>,
     <implementedBy __builtin__.object>,
     <InterfaceClass zope.interface.Interface>]

Hmmm. So I tried this on our *custom setup*, and got this::

    (Pdb) pp list(providedBy(ob).__sro__)
    [<implementedBy br.cms.folder.ContentContainerReadDirectory>,
     <InterfaceClass zope.filerepresentation.interfaces.IReadDirectory>,
     <InterfaceClass zope.app.container.interfaces.IReadContainer>,
     <InterfaceClass zope.app.container.interfaces.ISimpleReadContainer>,
     <InterfaceClass zope.app.container.interfaces.IItemContainer>,
     <InterfaceClass zope.interface.common.mapping.IEnumerableMapping>,
     <InterfaceClass zope.interface.common.mapping.IReadMapping>,
     <InterfaceClass zope.interface.common.mapping.IItemMapping>,
     <InterfaceClass zope.interface.Interface>,
     <implementedBy zope.app.folder.filerepresentation.ReadDirectory>,
     <implementedBy __builtin__.object>]

Well it turns out that there's one place wherein we diverge from the
base ReadDirectory: we include a ``zope.interface.implements``
declaration::

    class ContentContainerReadDirectory(ReadDirectory):
        implements(zope.filerepresentation.interfaces.IReadDirectory)

Between the screaming and the sobbing, I commented out the
``implements`` line and tried again::

    (Pdb) pp list(providedBy(ob).__sro__)
    [<implementedBy br.cms.folder.ContentContainerReadDirectory>,
     <implementedBy zope.app.folder.filerepresentation.ReadDirectory>,
     <implementedBy __builtin__.object>,
     <InterfaceClass zope.interface.Interface>]

And lo and behold, our custom setup worked::

    ftp> cd /customer/bogus
    550 /customer/bogus: No such file or directory.

So!!! This situation happened because our class had the audacity to
declare its instances' provided interface! The same interface that the
ZCML adapter declarations say they provide. It seems to change the
interface / specification order in a way that then elicits different
behavior. The FTPView commands are registered for IReadContainer,
which is the direct parent of IReadDirectory.. So when I had the
``implements`` line in our custom setup, we could change into those
non-existing directories since the IReadDirectory adapter of the
parent directory ('customer') was being returned by its own getitem
lookup; and by declaring implements directly, IReadContainer shows up
in the SRO, allowing Zope to go ahead and bind the 'ls' view to this
funky directory.

This seems MAJORLY broken: the only way to get the right behavior
(aside from patching / overriding getitem) is to NOT declare support
(in Python) for the interface my IReadDirectory adapters provide.