[Zope3-dev] Re: Google SoC Project
Stephan Richter
srichter at cosmos.phy.tufts.edu
Tue May 9 08:16:05 EDT 2006
On Tuesday 09 May 2006 07:22, Jim Fulton wrote:
> I guess we need to make this a priority for the next release.
>
> Python simply does not support a general robust reload, other than
> restart.
>
> I think that there are 2 ways we can make progress in this area:
>
> - Speed up restart. I think there are a lot of ways that restarts
> can be made faster:
>
> o Optimizae what we're doing now. I suspect that there are some
> opportunuties here.
I have applied for the SoC with a proposal to enhance ZCML. My proposal is
attached. It discusses some of the optimization options we talked of before.
If accepted I would work on this and the result would be, naturally, in the
next Zope 3 and 2 release.
Regards,
Stephan
--
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training
-------------- next part --------------
==============
Enhancing ZCML
==============
Name: Stephan Richter
E-Mail: srichter at cosmos.phy.tufts.edu
IRC Nickname: srichter
How much time do you expect to have for this project? Please list jobs,
summer classes, and/or vacations that you'll need to work around:
I am a Ph.D. student and do take any classes anymore. I am going to teach
the first summer session Physics 11 (Calculus-based mechanics) and continue
my thesis. Overall I think 3-4 weeks of time during the summer are a
realistic estimate.
Development experience:
- 9 years of Python experience.
- 6 years of Zope experience.
- Zope 3 core developer and release manager.
Please describe your usage experience/familiarity with the project you are
applying for:
I am a Zope 3 core developer. I am planning to implement new features in
Zope 3 that will make it easier to develop highly-customizable content
management systems, like Plone, using Zope 3.
What school do you attend? How many years have you attended there? What is
your specialty/major at the school?
Tufts University, Somerville, MA. Ph.D. in Physics, started in 2002. I am
developing models to simulate the immune system repsonse upon SIV/HIV
infections.
Project Details:
Project Outline
---------------
Zope 3 uses a component architecture as one of its most basic building
blocks. Since Zope 3 is a Web Application server, it must be possible to run
multiple Web sites on one application instance. The consequence is that Zope
must be configurable on a site by site basis. Thus an extension of the
component architecture allows us to define components globally (for all
sites) and locally (site specific).
Global components are commonly configured using an XML-based configuration
language called ZCML. Currently, local components can only be created and
configured via the Web UI. The advantage of doing so is that local
components can store their state in the ZODB. But this also means that it is
very difficult and cumbersome to register local components using regular
filesystem-based code. Clearly, this functionality is
sub-optimal. Oftentimes you want to be able to define site-specific (local)
components from the filesystem. This is particularly true for presentation
code, where it is often not a requirement that the state must be stored in
the ZODB.
I propose to allow local components to be configurable through ZCML. This
goal became feasible with the recent component architecture refactorings by
Jim Fulton. Any site (local or global) can now have a set of base sites that
are used to provide additional components. I want to allow ZCML to specify
any number of base sites and add components to it. Here is an example of how
I imagine it to look like (ZCML)::
<site name="my-base-site" />
<configure use-site="my-base-site">
...
</configure>
The new ``site`` directive creates a new site. The ``configure`` directive
will grow an ``use-site`` attribute that specifies the site to put in the
components. By default, ``use-site`` will use the global site. This also
ensures full backward-compatibility.
There will be a new registry of all ZCML-defined sites. All existing ZCML
directives have to be reviewed and it must be ensured that they are
multi-site aware. The tricky part of the implementation will be to hook in
those sites as bases to local sites. It must be ensured that the ZODB can
load having filesystem-based sub-sites, error handling must be carefully
considered and an UI must be written.
I believe that this feature is essential for highly customizable
applications liek Plone. It will allow skinning of sites using
filesystem-based development, something several people in the Plone
community have strived for for a long time. I think once the community
starts to discover the implications of this development, many new
possibilities will open.
Another big problem in Zope 3 is the startup time. Some code profiling has
determined that most of the time in the startup process is lost in parsing,
converting and validating ZCML directives with their schemas. Thus, this
startup problem is not purely a Zope 3 problem, but one that affects
everyone using ZCML.
This problem can be addressed in several ways. The most obvious one to Zope
2 developers would be not to restart the application server, but only reload
the packages and modules that were affected by the code changes. This
approach has been used in Zope 2 for many years, but it several serious
problems and some of the smartest people I know have not been able to
completely solve the problems. Based on that, I do not think that a proposal
suggesting this approach would be accepted.
The second approach is to reduce the ZCML processing time, which could be
integrated into the reload mechanism for Zope 2. This can be accomplished by
storing some binary representation of the ZCML, similarly to ``*.pyc`` files
in Python. Again there are several choices to consider and they should
probably all be tried. The first solution would be to store a pickle of each
parsed directive, namely the action and its arguments. There would be one
pickle file fore each ZCML file. When the ZCML file changed, the pickle
would be updated. Pickle loading would be much faster than pure ZCML
loading, since no XML-parsing, value conversion and schema validation would
be necessary.
On the other hand, ZCML creates actions that are eventually
executed. Actions are created by executing the directive handlers. Thus the
optimization in this approach would be even greater. The problem with this
approach is that not all directives are easily pickable. Directive handlers
often create new types/classes on the fly. This problem could be solved by
ensuring that directives only create pickable actions. Clearly, this would
require a lot more work, since you would have to go through all directives
to ensure their pickability and also provide fallbacks for 3rd-party
directives.
Overall, I believe that the performance of ZCML could be improved
drastically either way and that the benefit would be immediately visible for
anyone using ZCML, including Zope 2 developers using Zope 3 components via
Five.
All contributions will be made to the Zope 3 SVN core and will be licensed
under ZPL 2.1, a GPL-compatible license. If truly completed by September 1,
2006 all features will be part of the Zope 3.4 and 2.11 release. Special
thanks and credit will be given to the Google SoC Program and the Plone
Foundation for sponsoring this development.
Time Estiamtes
--------------
3 days: Writing a proposal for each feature. Zope 3 uses a proposal-based
development process and no feature can be checked into the trunk
without an accepted proposal.
2 days: Proposal discussion(s).
7 days: Implementation of the local sites through ZCML proposal. This includes
extensive testing and documentation as required by the Zope 3
development process.
15 days: Research and implementation of the ZCML optimizations. This
includes extensive testing, documentation and profiling.
More information about the Zope3-dev
mailing list