[Zope3-dev] Re: Google SoC Project

[Jim Fulton]

Stephan Richter wrote:
> On Tuesday 09 May 2006 07:22, Jim Fulton wrote:
> 
>>I guess we need to make this a priority for the next release.
>>
>>Python simply does not support a general robust reload, other than
>>restart.
>>
>>I think that there are 2 ways we can make progress in this area:
>>
>>- Speed up restart.  I think there are a lot of ways that restarts
>>   can be made faster:
>>
>>   o Optimizae what we're doing now.  I suspect that there are some
>>     opportunuties here.
> 
> 
> I have applied for the SoC with a proposal to enhance ZCML. My proposal is 
> attached. It discusses some of the optimization options we talked of before. 

...

>   I propose to allow local components to be configurable through ZCML. This
>   goal became feasible with the recent component architecture refactorings by
>   Jim Fulton. Any site (local or global) can now have a set of base sites that
>   are used to provide additional components. I want to allow ZCML to specify
>   any number of base sites and add components to it. Here is an example of how
>   I imagine it to look like (ZCML)::
> 
>     <site name="my-base-site" />

I'll note that I'd like site-name to be a convenience directive that is
equivalent to defining a GlobalComponents object and registering it as
an IComponents utility.  Note that GlobalComponents doesn't exist yet. :)
It should be very similar to BaseGlobalComponents, except that it pickles
and unpickles as a named utilities registered with a BaseGlobalComponents.


>     <configure use-site="my-base-site">
>       ...
>     </configure>
> 
>   The new ``site`` directive creates a new site. The ``configure`` directive
>   will grow an ``use-site`` attribute that specifies the site to put in the
>   components. By default,

Actually, the site to put the components in. :)

>   There will be a new registry of all ZCML-defined sites.

No!  Use the base utility registry.

 >   All existing ZCML
>   directives have to be reviewed and it must be ensured that they are
>   multi-site aware. The tricky part of the implementation will be to hook in
>   those sites as bases to local sites. It must be ensured that the ZODB can
>   load having filesystem-based sub-sites, error handling must be carefully
>   considered and an UI must be written.

The pickling aspects are pretty trivial.  I'm not sure what UI you are refering to.

...

>   Another big problem in Zope 3 is the startup time. Some code profiling has
>   determined that most of the time in the startup process is lost in parsing,
>   converting and validating ZCML directives with their schemas. Thus, this
>   startup problem is not purely a Zope 3 problem, but one that affects
>   everyone using ZCML.
> 
>   This problem can be addressed in several ways. The most obvious one to Zope
>   2 developers would be not to restart the application server, but only reload
>   the packages and modules that were affected by the code changes. This
>   approach has been used in Zope 2 for many years, but it several serious
>   problems and some of the smartest people I know have not been able to
>   completely solve the problems. Based on that, I do not think that a proposal
>   suggesting this approach would be accepted.
> 
>   The second approach is to reduce the ZCML processing time, which could be
>   integrated into the reload mechanism for Zope 2. This can be accomplished by
>   storing some binary representation of the ZCML, similarly to ``*.pyc`` files
>   in Python. Again there are several choices to consider and they should
>   probably all be tried. The first solution would be to store a pickle of each
>   parsed directive, namely the action and its arguments. There would be one
>   pickle file fore each ZCML file. When the ZCML file changed, the pickle
>   would be updated. Pickle loading would be much faster than pure ZCML
>   loading, since no XML-parsing, value conversion and schema validation would
>   be necessary.

Note that this will require a refactoring of ZCML handlers to define picklable
actions.  This will also require refactoring so that work now done by handlers
be defered to action execution.

>   On the other hand, ZCML creates actions that are eventually
>   executed. Actions are created by executing the directive handlers. Thus the
>   optimization in this approach would be even greater. The problem with this
>   approach is that not all directives are easily pickable. Directive handlers
>   often create new types/classes on the fly. This problem could be solved by
>   ensuring that directives only create pickable actions. Clearly, this would
>   require a lot more work, since you would have to go through all directives
>   to ensure their pickability and also provide fallbacks for 3rd-party
>   directives.

Yup.  One possibility is to have a mechanism in which the pickle files
are created only when possible.  That is, we try to pickle the actions
and give up if pickling fails. If we update the core directives to
work this way, that will account for most configuration files and
could provide a significant speedup even if not all configuration files
are handled.  Note that, if this is pursued. then a prototype should
test the speedup by updating some core directives and measuring the
impact.

>   Overall, I believe that the performance of ZCML could be improved
>   drastically either way and that the benefit would be immediately visible for
>   anyone using ZCML, including Zope 2 developers using Zope 3 components via
>   Five.

IMO, the action pickling work should be deferred until we allow configuration
to be expressed in Python and assess the impacts.  If many people switch to
defining configuration actions in Python, then speeding action generation from
XML could be moot.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org