[Zope3-dev] a bug in the Zope 3 (and 2) page template engine

[Martijn Faassen]

Hi there,

I just tracked down what I believe is a bug in the Zope 3 page template 
engine. I didn't know it was a bug in the Zope 3 page template engine as 
I was tracking down a bug in Silva with Zope 2.10, which of course uses 
the Zope 3 page template engine. For the proposed solution, see the 
bottom of the mail.

Let's consider the following code fragment:

<img tal:replace="structure python:'foo'" tal:attributes="title 
python:'bar'" />

In Zope 2 before Zope 2.10, this line works without error. The 
tal:attributes line of course is pretty useless, as the whole img tag 
gets replaced anyway, but it doesn't bail out with an error.

In Zope 2.10 and Zope 3.3, this gives us the following error [taken from 
Zope 3]:

   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 271, in __call__
     self.interpret(self.program)
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 346, in interpret
     handlers[opcode](self, args)
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 534, in do_optTag_tal
     self.no_tag(stuff[-2], stuff[-1])
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 516, in no_tag
     self.interpret(program)
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 346, in interpret
     handlers[opcode](self, args)
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 760, in do_insertStructure_tal
     self.insertHTMLStructure(text, repldict)
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 784, in insertHTMLStructure
     gen = AltTALGenerator(repldict, self.engine, 0)
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talinterpreter.py", 
line 65, in __init__
     TALGenerator.__init__(self, expressionCompiler, xml)
   File 
"/home/faassen/buildout/z331/lib/python/zope/tal/talgenerator.py", line 
41, in __init__
     self.CompilerError = expressionCompiler.getCompilerError()
AttributeError: 'ZopeContext' object has no attribute 'getCompilerError'

The difference between Zope 2 and Zope 3 is that ZopeContext is defined 
somewhere else, but that is irrelevant to this error.

Let's take a look at talgenerator.py, line 41:

         self.CompilerError = expressionCompiler.getCompilerError()

An AttributeError occurs here as getCompilerError does not exist. That's 
correct, as ZopeContext doesn't define one, nor should it. We should've 
gotten a ZopeEngine instance here, and in other code paths, that's what 
we get. This provides the interface ITALExpressionCompiler, which has 
getCompilerError.

What is calling talgenerator with the wrong argument for 
expressionCompiler? Let's go to talinterpreter, around line 784:

     def insertHTMLStructure(self, text, repldict):
         from zope.tal.htmltalparser import HTMLTALParser
         gen = AltTALGenerator(repldict, self.engine, 0)

That looks innocent enough. 'self.engine' is passed into AltTALGenerator 
(which in turns ends up in __init__ of TALGenerator). Unfortunately 
self.engine is very very badly named and is in fact a ZopeContext instance!

Why does this bug not happen more often? Because insertHTMLStructure is 
*only* called if tal:attributes there. Let's go to the relevant section, 
do_insertStructure_tal in talinterpreter.py:

     def do_insertStructure_tal(self, (expr, repldict, block)):
         ...
         if not (repldict or self.strictinsert):
             # Take a shortcut, no error checking
             self.stream_write(text)
             return
         if self.html:
             self.insertHTMLStructure(text, repldict)
         else:
             self.insertXMLStructure(text, repldict)

What is going on here is that normally, if no repldict exists (this is 
generated by tal:attributes), stream_write(text) is called. This works 
just fine.

In the case where there *is* a repldict, the broken version of 
insertHTMLStructure is called, leading to this error.

Now what's the hacky fix? It turns out that self.engine has an attribute 
_engine, that *is* the expression compiler. We can therefore fix the 
broken insertHTMLStructure like this:

     def insertHTMLStructure(self, text, repldict):
         from zope.tal.htmltalparser import HTMLTALParser
         gen = AltTALGenerator(repldict, self.engine._engine, 0)

That's ugly however. I looked at the old Zope 2 implementation, and that 
looks prettier:

     def insertHTMLStructure(self, text, repldict):
         from zope.tal.htmltalparser import HTMLTALParser
         gen = AltTALGenerator(repldict, self.engine.getCompiler(), 0)

Unfortunately, 'getCompiler' doesn't exist in Zope 3, so that won't work 
unless we add it again to the appropriate interface.

I imagine adding this back would be the best way to go. Anyone have any 
objections? I will go and fix this in Zope 3.3 and trunk. I will add a 
test. I will need to make sure the interface also has getCompiler() added.

This might also be a good occasion for a new Zope 2.10 bugfix release. 
Silva is running into this currently, though of course now that we know 
what the cause is we can work around it.

Regards,

Martijn