[Zope3-dev] Re: Using parent references rather than context
wrappers to represent containment
Jim Fulton
jim at zope.com
Mon Aug 11 07:10:26 EDT 2003
I failed to make an important point in my write up.
In my opinion, the #1 benefit of this change would
be to support direct references between objects without
losing information necessary to compute paths and URIs.
It allows us to avoid the complexity of using paths or
hub ids as described in section "Context wrappers complicate
object references".
This greatly simplifies modeling direct object associations and
implementing object relations.
Jim
Jim Fulton wrote:
>
> This would be a proposal in the Zope 3 Wiki, if the Zope 3 wiki was
> writable.
>
> Status: IsWorkInProgress
>
> Author
>
> JimFulton
>
> Problem
>
> Zope organizes objects into a containement hierarchy. Zope uses
> this organization to share information among objects, including:
>
> - Software and configuration settings
>
> - Security statements
>
> The sharing of information in a containment hierarchy mirrors the
> sharing of information of information on class hierarchies
> (directed acyclic graphs if we're being fussy).
>
> The use of containment hierarchies to help organize information
> has served Zope very well.
>
> The original technology for sharing information in containment
> hierarchies in Zope was Acquisition. Acquisition allowed
> attributes to be be looked up in containment hierarchies in much
> the same way that they are looked up in inheritance hierarchies.
> Acquisition was implemented using wrappers. An acquisition
> wrapper contains a reference to the object wrapped object
> ('aq_self') and to the object the wrapped was accessed through
> ('aq_parent'). A wrapper-based approach was used to avoid
> circular references. Acquisition was originally implemented for
> Python 1, which lacked a cyclic garbage collector.
>
> Zope's acquisition facility was highly automated. Acquisition
> wrappers were created whenever a Zope object was accessed as an
> attribute of another Zope object. Zope programmers rarely had to
> create acquisition wrappers manually. When an object method was
> called, the method was bound to the acquisition wrapper. Object
> methods had full access to acquisition. This high degree of
> automation made acquisition quite ubiquitious and and easy to use
> in Zope 2.
>
> Acquisition did have some disadvantages:
>
> - The commonly used form of acquistion was "implici"t acquisition.
> With implicit acquisition, a normal attribute access was
> satisfied through acquisition if it could not be satisfoed
> through inheritence. This was actually a feature most of the
> time. It mad efinding information easy. Unfortunately, it was
> too easy and powerful. You'd sometimes get information you
> didn't expect.
>
> - Wrappers obscure object identity and type. When wrappers are
> widely used, it becomes harder to test an object's identity
> because a wrapper has a different identity than the object it
> wraps. Similarly, whille wrappers have a different
> type. Software that introspects an object's type can be confused
> by wrappers.
>
> - Acquisition required a special meta class. The implementation
> of acquisition relied on a special meta class, Extension Class.
> This meta class was used throughout Zope for other reasons, such
> as providing type class unification and persistence, so this was
> not a burden in Zope 2.
>
> - Acquisition wrappers added a number of specialized attributes to
> the objects that they wrapped. Some of these, namely 'aq_self'
> and 'aq_base', introduced security problems.
>
> Zope 3 doesn't use Zope 2's acquisition mechanism for a number of
> reasons:
>
> - While Zope 3 still uses a containment hierarchy to share
> information, it uses far more explicit lookup algorithms:
>
> o An API call is required to look up an object using containment.
>
> o There are different APIs for looking up different kinds of
> objects, For example, services are looked up with a different
> API than views are looked up.
>
> - Zope 3 seaks to be as open and inclusive of other Python
> software as possible. We don't require Zope-specific mix-in
> classes. We don't require objects to implement specific APIs to
> be usable in Zope 3. We are certainly unwilling to require
> specialized meta classes.
>
> We are trying hard not to cause surprises for the Python
> programmer.
>
> Zope 3 does use a wrapper-based model for sharing information.
> Zope 3 has "context wrappers":
>
> - Wrappers are not created automatically when an attribute is
> accessed. Wrappers are only created automatically dureing URL
> traversal. They must be created manually at other times.
>
> To the degree that creation of wrappers is automated, we provide
> the automation externally to the wrapped objects.
>
> - Wrappers don't provide any attributes that can be accessed
> directly. Rather API's are used to access wrapper information.
>
> - When calling a method on a wrapped object, the method isn't
> bound to a wrapped object unless the method is marked as a
> context-aware method.
>
> Difficult to maintain context information
>
> Zope 3 context wrappers are far more explicit that Zope 2
> acquisition wrappers, or, in other words, Zope 3 context wrappers
> are far less automated than Zope 2 acquisition wrappers. This
> presents some risk. There is a danger that Zope 3 code will
> become littered with excess context-wrapper manipulation code.
>
> Early on, it appeared that creating context wrappers through
> traversal would be sufficient. Views and adapters, created
> externally to objects were given context wrapped objects and,
> this, have full and sufficiently automatic access to context
> information. As Zope 3 software has become more sophisticated,
> however, the level of automation provided by context wrappers
> seems to be less adequate:
>
> - More sophisticated and more zope-specific content objects need
> access to context information. This has been especially true of
> "meta content" objects, like services and the objects that
> support them.
>
> - As object's became more sophisticated other access meachanisms
> other than URL traversal became common. When objects are
> accessed in other ways, context wrappers aren't created
> automatically. Access methods must create the wrappers, which
> can become quite burdonsome.
>
> An added challenge is that context management must be complete.
> Context wrappers must be maintained throughout the
> chain. Consider three objects, A, B and C, such that B is
> contained in A and C is contained in B. Suppose we access B
> through A and C through B. If a context wrapper is created for
> C in B but not for B in A, many context dependent operations on
> the resulting context-wrapped C will fail.
>
> Bugs in context management are hard to debug and are
> increasingly common.
>
> Context wrappers complicate object references
>
> Containment is just one of many interesting relationships that
> can exist between objects. We often need to model other forms
> of relationship between objects. There are two common ways to
> think about this:
>
> - Direct associations between objects. Associations are model
> as direct references between objects. Containment is one form
> of association.
>
> We don't want to mix up containment with other associations.
> In Zope 2's Acquisition model, all associations were treated
> as containment. To avoid this, we made it difficult to store
> references to objects obtained from elsewhere by making.
>
> - Relations betweed objects. Relations model relationships
> between objects externally to the objects. A "relation" is a
> set of such relations. Each entry in the relation is a "tuple"
> (not to be confused with Python tuples) containing the related
> objects. For example, we might have a table that models a
> relation. Each entry in the table includes references to the
> related objects along with additional data about the
> relationship.
>
> Historically, we tried to prevent storing acqusition wrappers in
> object attributes to prevent circular references. We did this
> by making acquisition wrappers non picklable. Thus, an error
> would occur if someone tried to store an acquisition wrapper in
> a persistent object. We retained this limitation in context
> wrappers.
>
> The inability to implement non-containment relationships through
> direct references between objects has led to the use of object
> identifiers of various forms. Objects don't refer to other
> object's directly. Rather, objects store identifiers of objects
> they refer to. When they want to get to the referenced object,
> they must somehow evaluate the identifier.
>
> The most common form of identifier is an object path. Such
> references are similar to unix symbolic links and have the same
> strengths and weaknesses. References can become stale if a
> referenced object is moved or deleted.
>
> An alternative to storing paths directly is to use an object
> directory, called an "object hub" to keep track of object
> locations. Refering objects store a hub ID. When they need to
> accessed a referenced object, they ask the hub to get the
> objects associated with a hub id. The hub looks up the object
> path and used the path to look up the object. The hub has the
> responsibility for tracking the current location of the
> object. With a suitable notification system, the hub can keep
> the location of the object up to date. In this way, an object
> reference is not broken when an object is moved, although it may
> still be broken if an object is deleted.
>
> The use of indirect references betwen objects introduces
> referential integrity issues with solutions that are similar to
> those found in relational databases.
>
> Certainly, the use of indirect references introduce significant
> complexity and overhead into applications. Direct references
> would be simpler and more efficient in many ways.
>
> Fortunately, with the advent of Python 2, we are free to create
> circular references between objects. Python's cyclic garbage
> collector will still be able to free unuded objects.
>
> Note that ZODB has always mitigated circular reference problems
> among saved persistent objects. Cycles are broken when
> participating objects are deactivated after periods of unuse.
> Cycles are not broken among objects that haven't been stored in
> the database, however, and among non-persistent objects.
>
> Also note that new ZODB storages will benefit from a lack of
> cycles among objects. These newer storages use reference
> counting to speed detection of objects that can be removed from
> the database, obviating the need for heavy pack operations. For
> this reason, avoidence of large amounts of cyclic garbage is
> worthwhile. This is why we have gone to great lengths to avoid
> cycles in our BTree implementation.
>
> Parent references considered desireable
>
> It would appear that a simpler and more direct approach to
> maintaining containment relationships is to actuallt store
> references to containers in contained objects. I've considered
> this for some time. Guido van Rossom suggested it on a number of
> occasions. Recently, Phillip Eby suggested it on the zope3-dev
> mailing list:
>
> http://mail.zope.org/pipermail/zope3-dev/2003-June/007455.html
>
> This spured much comment favorable to at least trying this
> approach.
>
> Using parent references has several obvious issues:
>
> - It introduces circular references. As noted above, this is not
> much of an issue, especially for Python 2.
>
> - It makes it impossible for an object to be contained in
> multiple containers. After some debate, it appears that this
> is a feature. In general, people don't really want an object
> to have multiple containers. People do want to be able to
> reference an object from multiple places, but these references
> need not be containment references. There is a strong desire
> to be able to mirror an object system to a file-system through
> various forms of file-system synchronization or
> representation. Because file-stems are hierarchical,
> synchronization is simpler if there if the object system is
> hierarchical. Containment relationships provide a basis for
> such a hierarchy. (Conceiveably, applications could employ
> additional or alternative hierarchies.
>
> - The most significant problem with requiring parent references
> is that it adds a requirement for objects to be used in Zope. I
> *really* don't want to require zope-specific interfaces for an
> object to be usable in Zope. It appears that this problem can
> be mitigated using decorators that wrap objects and provide
> the extra information. Unlike acquisition and context
> wrappers, these decorators must be picklable, as they are
> stored in referncing objects.
>
> Prototyping parent references
>
> To pursue the idea of using direct parent references instead of
> context wrappers to represent containment relationships, a
> prototype effort is underway. This prototype effort has
> uncovered additional issues that need to be resolved.
>
> Progress to date
>
> To begin with, an IContained interface was defined::
>
> class IContained(Interface):
> """Objects contained in containers
> """
>
> __container__ = Attribute("The container")
>
> __name__ = schema.TextLine(
> __doc__=
> """The name within the container
>
> The container can be traversed with this name to get
> the object. """)
>
> Three implementations of this interface were provided:
>
> - A (trivial) mix-in class for content objects that need to
> access their containment context.
>
> - A decorator that can wrap non-context-aware objects. The
> decorator manages the '__container__' and '__name__'
> attributes for objects that haven't been modified to support
> 'IContained'.
>
> - Context wrappers were extended to implement 'IContained'.
>
> Having context wrappers implements `IContained` allows the
> interface to be used to introspect the containement relationship
> regardless of whether it is implemented through direct
> references or through context wrappers, Anything that needs to
> search containment context uses this common interface.
>
> All code that searches context has been modified to use the
> IContained interface. Doing so has revealed a number of issues.
>
> There are facilities for supporting context sensitivity through
> context-aware methods. These facilities are unnecessary with
> direct parent references. Classes for context-aware context
> objects were modified to implement 'IContained' and were
> modified to not use context methods.
>
> Note that, context wrappers are still used for objects that are
> traversed to, but not stored, most notably, for views.
>
> Issues uncovered so far
>
> A number of issues have been uncovered by the prototype effort to
> date.
>
> 1. It isn't enough for contained objects to implement IContained.
> Containing objects need to manage the '__container__' and
> '__name__' attributes. In addition, containers need to assure
> that contained items implement 'IContained', wrapping objects
> in contained proxies as necessary. As with context wrappers,
> it's essential that this be done consistently. It's not enough
> for a container to satisfy this requirement if one if it's
> containers does not.
>
> It appears that 'IContainer' should be expanded to collaborate
> with 'IContained'. At first, this seems a bit onerous,
> however, 'IContainer' is already a Zope-specific interface, so
> perhaps this isn't such a problem.
>
> 2. Context wrappers have evolved to provide more than context
> management. They provide dictionaries that can be used for
> caching. They also provide support for context aware
> decoration. When we create a context wrapper for an object, we
> do so through adaptation. Specialized adapters for containers
> provide support for generating expected events when containers
> are modified.
>
> With the move toward parent references, context wrappers are
> not used in many places there were used before. It no longer
> seems appropriate to rely on context decorators to ensure that
> necessary modification events are generated.
>
> Three posible ways to generate modification events are:
>
> a. Move the responsability for generating modification events
> back to applicatin code (views or or adapters used by views).
>
> b. Add the responsability for for generating modification
> events to IContainer. This begins to make 'IContainer' a much
> richer interface than might be appropriate. There might be
> some interest in using the containment framework outside of
> Zope, where requiring Zope's event framework might be a
> liability.
>
> c. Continue the use of a decorator model, but decouple
> decorating for the purpose of supporting the Zope application
> framework from decoration to support context management.
>
> 3. It would be useful to unify the context managemnt used by views
> and the 'IContained' framework. It would simplify matters if
> one accessed the context of a view the same way one accessed an
> object's container. This would avoid the need to put context
> wrappers around views.
>
> Right now, we require views to provide a context attribute.
>
> We could:
>
> a. Optionally allow views to implement 'IContained'.
> We could encourage this, for example, by modifying
> 'BrowserView' to implement 'IContained'. Views would still
> be required to provide a context attribute.
>
> b. Require views to implement 'IContained'. At this point, it
> wouldn't make sense to require a 'context' attribute.
>
> If we do this, does it makse sense to name the attribute
> '__container__'? Would '__parent__' be better?
> '__context__'? Would we need to rename the the assiated ZPT
> variable?
>
> 4. The attributes defined by 'IContained' need to accessable
> through security proxies. I think that these attributes should
> be unconditionally accessable in all objects.
>
>
> Comment desired!
>
> I'd really like to get comment on the issues above. Absent,
> comment, here is how I'm thinking of resolving these issues:
>
> 1. Add responsability for collaborating with 'IContained' to
> 'IContainer'.
>
> 2. Move responsability for generating modification events for
> containers back to application code.
>
> 3. Optionally allow vies wo implement 'IContained' and add
> 'IContained' support to BrowserView.
>
> 4. Modify the security framework to allow access to
> '__container__' and '__name__'.
>
> Jim
>
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (703) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
More information about the Zope3-dev
mailing list