[Zope3-dev] Re: Google SoC Project

Stephan Richter srichter at cosmos.phy.tufts.edu
Tue May 9 08:16:05 EDT 2006

On Tuesday 09 May 2006 07:22, Jim Fulton wrote:
> I guess we need to make this a priority for the next release.
> Python simply does not support a general robust reload, other than
> restart.
> I think that there are 2 ways we can make progress in this area:
> - Speed up restart.  I think there are a lot of ways that restarts
>    can be made faster:
>    o Optimizae what we're doing now.  I suspect that there are some
>      opportunuties here.

I have applied for the SoC with a proposal to enhance ZCML. My proposal is 
attached. It discusses some of the optimization options we talked of before. 
If accepted I would work on this and the result would be, naturally, in the 
next Zope 3 and 2 release.

Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training
-------------- next part --------------
Enhancing ZCML

Name: Stephan Richter
E-Mail: srichter at cosmos.phy.tufts.edu
IRC Nickname: srichter

How much time do you expect to have for this project? Please list jobs,
summer classes, and/or vacations that you'll need to work around:

  I am a Ph.D. student and do take any classes anymore. I am going to teach
  the first summer session Physics 11 (Calculus-based mechanics) and continue
  my thesis. Overall I think 3-4 weeks of time during the summer are a
  realistic estimate.

Development experience:

  - 9 years of Python experience.
  - 6 years of Zope experience.
  - Zope 3 core developer and release manager.

Please describe your usage experience/familiarity with the project you are
applying for:

  I am a Zope 3 core developer. I am planning to implement new features in
  Zope 3 that will make it easier to develop highly-customizable content
  management systems, like Plone, using Zope 3.

What school do you attend? How many years have you attended there? What is
your specialty/major at the school?

  Tufts University, Somerville, MA. Ph.D. in Physics, started in 2002. I am
  developing models to simulate the immune system repsonse upon SIV/HIV

Project Details:

  Project Outline

  Zope 3 uses a component architecture as one of its most basic building
  blocks. Since Zope 3 is a Web Application server, it must be possible to run
  multiple Web sites on one application instance. The consequence is that Zope
  must be configurable on a site by site basis. Thus an extension of the
  component architecture allows us to define components globally (for all
  sites) and locally (site specific).

  Global components are commonly configured using an XML-based configuration
  language called ZCML. Currently, local components can only be created and
  configured via the Web UI. The advantage of doing so is that local
  components can store their state in the ZODB. But this also means that it is
  very difficult and cumbersome to register local components using regular
  filesystem-based code. Clearly, this functionality is
  sub-optimal. Oftentimes you want to be able to define site-specific (local)
  components from the filesystem. This is particularly true for presentation
  code, where it is often not a requirement that the state must be stored in
  the ZODB.

  I propose to allow local components to be configurable through ZCML. This
  goal became feasible with the recent component architecture refactorings by
  Jim Fulton. Any site (local or global) can now have a set of base sites that
  are used to provide additional components. I want to allow ZCML to specify
  any number of base sites and add components to it. Here is an example of how
  I imagine it to look like (ZCML)::

    <site name="my-base-site" />

    <configure use-site="my-base-site">

  The new ``site`` directive creates a new site. The ``configure`` directive
  will grow an ``use-site`` attribute that specifies the site to put in the
  components. By default, ``use-site`` will use the global site. This also
  ensures full backward-compatibility.

  There will be a new registry of all ZCML-defined sites. All existing ZCML
  directives have to be reviewed and it must be ensured that they are
  multi-site aware. The tricky part of the implementation will be to hook in
  those sites as bases to local sites. It must be ensured that the ZODB can
  load having filesystem-based sub-sites, error handling must be carefully
  considered and an UI must be written.

  I believe that this feature is essential for highly customizable
  applications liek Plone. It will allow skinning of sites using
  filesystem-based development, something several people in the Plone
  community have strived for for a long time. I think once the community
  starts to discover the implications of this development, many new
  possibilities will open.

  Another big problem in Zope 3 is the startup time. Some code profiling has
  determined that most of the time in the startup process is lost in parsing,
  converting and validating ZCML directives with their schemas. Thus, this
  startup problem is not purely a Zope 3 problem, but one that affects
  everyone using ZCML.

  This problem can be addressed in several ways. The most obvious one to Zope
  2 developers would be not to restart the application server, but only reload
  the packages and modules that were affected by the code changes. This
  approach has been used in Zope 2 for many years, but it several serious
  problems and some of the smartest people I know have not been able to
  completely solve the problems. Based on that, I do not think that a proposal
  suggesting this approach would be accepted.

  The second approach is to reduce the ZCML processing time, which could be
  integrated into the reload mechanism for Zope 2. This can be accomplished by
  storing some binary representation of the ZCML, similarly to ``*.pyc`` files
  in Python. Again there are several choices to consider and they should
  probably all be tried. The first solution would be to store a pickle of each
  parsed directive, namely the action and its arguments. There would be one
  pickle file fore each ZCML file. When the ZCML file changed, the pickle
  would be updated. Pickle loading would be much faster than pure ZCML
  loading, since no XML-parsing, value conversion and schema validation would
  be necessary.

  On the other hand, ZCML creates actions that are eventually
  executed. Actions are created by executing the directive handlers. Thus the
  optimization in this approach would be even greater. The problem with this
  approach is that not all directives are easily pickable. Directive handlers
  often create new types/classes on the fly. This problem could be solved by
  ensuring that directives only create pickable actions. Clearly, this would
  require a lot more work, since you would have to go through all directives
  to ensure their pickability and also provide fallbacks for 3rd-party

  Overall, I believe that the performance of ZCML could be improved
  drastically either way and that the benefit would be immediately visible for
  anyone using ZCML, including Zope 2 developers using Zope 3 components via

  All contributions will be made to the Zope 3 SVN core and will be licensed
  under ZPL 2.1, a GPL-compatible license. If truly completed by September 1,
  2006 all features will be part of the Zope 3.4 and 2.11 release. Special
  thanks and credit will be given to the Google SoC Program and the Plone
  Foundation for sponsoring this development.

  Time Estiamtes

  3 days: Writing a proposal for each feature. Zope 3 uses a proposal-based
          development process and no feature can be checked into the trunk
          without an accepted proposal.

  2 days: Proposal discussion(s).

  7 days: Implementation of the local sites through ZCML proposal. This includes
          extensive testing and documentation as required by the Zope 3
          development process.

  15 days: Research and implementation of the ZCML optimizations. This
           includes extensive testing, documentation and profiling.

More information about the Zope3-dev mailing list