[Zope3-dev] Using parent references rather than context wrappers to
represent containment
Jim Fulton
jim at zope.com
Sun Aug 10 11:22:23 EDT 2003
This would be a proposal in the Zope 3 Wiki, if the Zope 3 wiki was writable.
Status: IsWorkInProgress
Author
JimFulton
Problem
Zope organizes objects into a containement hierarchy. Zope uses
this organization to share information among objects, including:
- Software and configuration settings
- Security statements
The sharing of information in a containment hierarchy mirrors the
sharing of information of information on class hierarchies
(directed acyclic graphs if we're being fussy).
The use of containment hierarchies to help organize information
has served Zope very well.
The original technology for sharing information in containment
hierarchies in Zope was Acquisition. Acquisition allowed
attributes to be be looked up in containment hierarchies in much
the same way that they are looked up in inheritance hierarchies.
Acquisition was implemented using wrappers. An acquisition
wrapper contains a reference to the object wrapped object
('aq_self') and to the object the wrapped was accessed through
('aq_parent'). A wrapper-based approach was used to avoid
circular references. Acquisition was originally implemented for
Python 1, which lacked a cyclic garbage collector.
Zope's acquisition facility was highly automated. Acquisition
wrappers were created whenever a Zope object was accessed as an
attribute of another Zope object. Zope programmers rarely had to
create acquisition wrappers manually. When an object method was
called, the method was bound to the acquisition wrapper. Object
methods had full access to acquisition. This high degree of
automation made acquisition quite ubiquitious and and easy to use
in Zope 2.
Acquisition did have some disadvantages:
- The commonly used form of acquistion was "implici"t acquisition.
With implicit acquisition, a normal attribute access was
satisfied through acquisition if it could not be satisfoed
through inheritence. This was actually a feature most of the
time. It mad efinding information easy. Unfortunately, it was
too easy and powerful. You'd sometimes get information you
didn't expect.
- Wrappers obscure object identity and type. When wrappers are
widely used, it becomes harder to test an object's identity
because a wrapper has a different identity than the object it
wraps. Similarly, whille wrappers have a different
type. Software that introspects an object's type can be confused
by wrappers.
- Acquisition required a special meta class. The implementation
of acquisition relied on a special meta class, Extension Class.
This meta class was used throughout Zope for other reasons, such
as providing type class unification and persistence, so this was
not a burden in Zope 2.
- Acquisition wrappers added a number of specialized attributes to
the objects that they wrapped. Some of these, namely 'aq_self'
and 'aq_base', introduced security problems.
Zope 3 doesn't use Zope 2's acquisition mechanism for a number of
reasons:
- While Zope 3 still uses a containment hierarchy to share
information, it uses far more explicit lookup algorithms:
o An API call is required to look up an object using containment.
o There are different APIs for looking up different kinds of
objects, For example, services are looked up with a different
API than views are looked up.
- Zope 3 seaks to be as open and inclusive of other Python
software as possible. We don't require Zope-specific mix-in
classes. We don't require objects to implement specific APIs to
be usable in Zope 3. We are certainly unwilling to require
specialized meta classes.
We are trying hard not to cause surprises for the Python
programmer.
Zope 3 does use a wrapper-based model for sharing information.
Zope 3 has "context wrappers":
- Wrappers are not created automatically when an attribute is
accessed. Wrappers are only created automatically dureing URL
traversal. They must be created manually at other times.
To the degree that creation of wrappers is automated, we provide
the automation externally to the wrapped objects.
- Wrappers don't provide any attributes that can be accessed
directly. Rather API's are used to access wrapper information.
- When calling a method on a wrapped object, the method isn't
bound to a wrapped object unless the method is marked as a
context-aware method.
Difficult to maintain context information
Zope 3 context wrappers are far more explicit that Zope 2
acquisition wrappers, or, in other words, Zope 3 context wrappers
are far less automated than Zope 2 acquisition wrappers. This
presents some risk. There is a danger that Zope 3 code will
become littered with excess context-wrapper manipulation code.
Early on, it appeared that creating context wrappers through
traversal would be sufficient. Views and adapters, created
externally to objects were given context wrapped objects and,
this, have full and sufficiently automatic access to context
information. As Zope 3 software has become more sophisticated,
however, the level of automation provided by context wrappers
seems to be less adequate:
- More sophisticated and more zope-specific content objects need
access to context information. This has been especially true of
"meta content" objects, like services and the objects that
support them.
- As object's became more sophisticated other access meachanisms
other than URL traversal became common. When objects are
accessed in other ways, context wrappers aren't created
automatically. Access methods must create the wrappers, which
can become quite burdonsome.
An added challenge is that context management must be complete.
Context wrappers must be maintained throughout the
chain. Consider three objects, A, B and C, such that B is
contained in A and C is contained in B. Suppose we access B
through A and C through B. If a context wrapper is created for
C in B but not for B in A, many context dependent operations on
the resulting context-wrapped C will fail.
Bugs in context management are hard to debug and are
increasingly common.
Context wrappers complicate object references
Containment is just one of many interesting relationships that
can exist between objects. We often need to model other forms
of relationship between objects. There are two common ways to
think about this:
- Direct associations between objects. Associations are model
as direct references between objects. Containment is one form
of association.
We don't want to mix up containment with other associations.
In Zope 2's Acquisition model, all associations were treated
as containment. To avoid this, we made it difficult to store
references to objects obtained from elsewhere by making.
- Relations betweed objects. Relations model relationships
between objects externally to the objects. A "relation" is a
set of such relations. Each entry in the relation is a "tuple"
(not to be confused with Python tuples) containing the related
objects. For example, we might have a table that models a
relation. Each entry in the table includes references to the
related objects along with additional data about the
relationship.
Historically, we tried to prevent storing acqusition wrappers in
object attributes to prevent circular references. We did this
by making acquisition wrappers non picklable. Thus, an error
would occur if someone tried to store an acquisition wrapper in
a persistent object. We retained this limitation in context
wrappers.
The inability to implement non-containment relationships through
direct references between objects has led to the use of object
identifiers of various forms. Objects don't refer to other
object's directly. Rather, objects store identifiers of objects
they refer to. When they want to get to the referenced object,
they must somehow evaluate the identifier.
The most common form of identifier is an object path. Such
references are similar to unix symbolic links and have the same
strengths and weaknesses. References can become stale if a
referenced object is moved or deleted.
An alternative to storing paths directly is to use an object
directory, called an "object hub" to keep track of object
locations. Refering objects store a hub ID. When they need to
accessed a referenced object, they ask the hub to get the
objects associated with a hub id. The hub looks up the object
path and used the path to look up the object. The hub has the
responsibility for tracking the current location of the
object. With a suitable notification system, the hub can keep
the location of the object up to date. In this way, an object
reference is not broken when an object is moved, although it may
still be broken if an object is deleted.
The use of indirect references betwen objects introduces
referential integrity issues with solutions that are similar to
those found in relational databases.
Certainly, the use of indirect references introduce significant
complexity and overhead into applications. Direct references
would be simpler and more efficient in many ways.
Fortunately, with the advent of Python 2, we are free to create
circular references between objects. Python's cyclic garbage
collector will still be able to free unuded objects.
Note that ZODB has always mitigated circular reference problems
among saved persistent objects. Cycles are broken when
participating objects are deactivated after periods of unuse.
Cycles are not broken among objects that haven't been stored in
the database, however, and among non-persistent objects.
Also note that new ZODB storages will benefit from a lack of
cycles among objects. These newer storages use reference
counting to speed detection of objects that can be removed from
the database, obviating the need for heavy pack operations. For
this reason, avoidence of large amounts of cyclic garbage is
worthwhile. This is why we have gone to great lengths to avoid
cycles in our BTree implementation.
Parent references considered desireable
It would appear that a simpler and more direct approach to
maintaining containment relationships is to actuallt store
references to containers in contained objects. I've considered
this for some time. Guido van Rossom suggested it on a number of
occasions. Recently, Phillip Eby suggested it on the zope3-dev
mailing list:
http://mail.zope.org/pipermail/zope3-dev/2003-June/007455.html
This spured much comment favorable to at least trying this
approach.
Using parent references has several obvious issues:
- It introduces circular references. As noted above, this is not
much of an issue, especially for Python 2.
- It makes it impossible for an object to be contained in
multiple containers. After some debate, it appears that this
is a feature. In general, people don't really want an object
to have multiple containers. People do want to be able to
reference an object from multiple places, but these references
need not be containment references. There is a strong desire
to be able to mirror an object system to a file-system through
various forms of file-system synchronization or
representation. Because file-stems are hierarchical,
synchronization is simpler if there if the object system is
hierarchical. Containment relationships provide a basis for
such a hierarchy. (Conceiveably, applications could employ
additional or alternative hierarchies.
- The most significant problem with requiring parent references
is that it adds a requirement for objects to be used in Zope. I
*really* don't want to require zope-specific interfaces for an
object to be usable in Zope. It appears that this problem can
be mitigated using decorators that wrap objects and provide
the extra information. Unlike acquisition and context
wrappers, these decorators must be picklable, as they are
stored in referncing objects.
Prototyping parent references
To pursue the idea of using direct parent references instead of
context wrappers to represent containment relationships, a
prototype effort is underway. This prototype effort has
uncovered additional issues that need to be resolved.
Progress to date
To begin with, an IContained interface was defined::
class IContained(Interface):
"""Objects contained in containers
"""
__container__ = Attribute("The container")
__name__ = schema.TextLine(
__doc__=
"""The name within the container
The container can be traversed with this name to get
the object. """)
Three implementations of this interface were provided:
- A (trivial) mix-in class for content objects that need to
access their containment context.
- A decorator that can wrap non-context-aware objects. The
decorator manages the '__container__' and '__name__'
attributes for objects that haven't been modified to support
'IContained'.
- Context wrappers were extended to implement 'IContained'.
Having context wrappers implements `IContained` allows the
interface to be used to introspect the containement relationship
regardless of whether it is implemented through direct
references or through context wrappers, Anything that needs to
search containment context uses this common interface.
All code that searches context has been modified to use the
IContained interface. Doing so has revealed a number of issues.
There are facilities for supporting context sensitivity through
context-aware methods. These facilities are unnecessary with
direct parent references. Classes for context-aware context
objects were modified to implement 'IContained' and were
modified to not use context methods.
Note that, context wrappers are still used for objects that are
traversed to, but not stored, most notably, for views.
Issues uncovered so far
A number of issues have been uncovered by the prototype effort to
date.
1. It isn't enough for contained objects to implement IContained.
Containing objects need to manage the '__container__' and
'__name__' attributes. In addition, containers need to assure
that contained items implement 'IContained', wrapping objects
in contained proxies as necessary. As with context wrappers,
it's essential that this be done consistently. It's not enough
for a container to satisfy this requirement if one if it's
containers does not.
It appears that 'IContainer' should be expanded to collaborate
with 'IContained'. At first, this seems a bit onerous,
however, 'IContainer' is already a Zope-specific interface, so
perhaps this isn't such a problem.
2. Context wrappers have evolved to provide more than context
management. They provide dictionaries that can be used for
caching. They also provide support for context aware
decoration. When we create a context wrapper for an object, we
do so through adaptation. Specialized adapters for containers
provide support for generating expected events when containers
are modified.
With the move toward parent references, context wrappers are
not used in many places there were used before. It no longer
seems appropriate to rely on context decorators to ensure that
necessary modification events are generated.
Three posible ways to generate modification events are:
a. Move the responsability for generating modification events
back to applicatin code (views or or adapters used by views).
b. Add the responsability for for generating modification
events to IContainer. This begins to make 'IContainer' a much
richer interface than might be appropriate. There might be
some interest in using the containment framework outside of
Zope, where requiring Zope's event framework might be a
liability.
c. Continue the use of a decorator model, but decouple
decorating for the purpose of supporting the Zope application
framework from decoration to support context management.
3. It would be useful to unify the context managemnt used by views
and the 'IContained' framework. It would simplify matters if
one accessed the context of a view the same way one accessed an
object's container. This would avoid the need to put context
wrappers around views.
Right now, we require views to provide a context attribute.
We could:
a. Optionally allow views to implement 'IContained'.
We could encourage this, for example, by modifying
'BrowserView' to implement 'IContained'. Views would still
be required to provide a context attribute.
b. Require views to implement 'IContained'. At this point, it
wouldn't make sense to require a 'context' attribute.
If we do this, does it makse sense to name the attribute
'__container__'? Would '__parent__' be better?
'__context__'? Would we need to rename the the assiated ZPT
variable?
4. The attributes defined by 'IContained' need to accessable
through security proxies. I think that these attributes should
be unconditionally accessable in all objects.
Comment desired!
I'd really like to get comment on the issues above. Absent,
comment, here is how I'm thinking of resolving these issues:
1. Add responsability for collaborating with 'IContained' to
'IContainer'.
2. Move responsability for generating modification events for
containers back to application code.
3. Optionally allow vies wo implement 'IContained' and add
'IContained' support to BrowserView.
4. Modify the security framework to allow access to
'__container__' and '__name__'.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (703) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
More information about the Zope3-dev
mailing list