[Zope-dev] (ZPatterns) Speeding up Skinscripts

[John Eikenberry]

Phillip J. Eby wrote:

> Congratulations, you're the first person (that I know of, anyway) to hit a
> wall here.  However, I think you may be barking up the wrong tree on your
> profiling.  See below.  

What a dubious honor.
 
> SkinScript expressions execute using Zope security checking.  If you're
> using a "WITH QUERY SQLmethod() COMPUTE attributelist" statement, security
> checks are applied to both the query expression and to every name
> referenced in 'attributelist'.  So the longer the attribute list, the
> slower the execution.  Although, now that I think of it, eval() is not used
> if you use a pure attribute name list like "name1,name2,name3", so the
> problem probably isn't on the attribute list side.  This makes it much less
> likely that the security machinery is to blame (unless you have a very slow
> user folder which also doesn't cache well, and you haven't given your
> SkinScript proxy roles to shortcut the security checks.)

Yeah... I just added proxy roles to all the skinscripts and it didn't make
that much of a difference (maybe a small one though).
 
> Now, for your SQL query...  is it by any chance dynamically generated in a
> complex way?  I mean, more complex than an sqlvar or two?  Keep in mind
> that when Zope caches SQL queries it does it by generating the text of the
> query and using it as a cache key.  That means any DTML in your query will
> be executed each time.  It'd have to be pretty complex, or else there'd
> need to be lots of slow security lookups, to get the kind of poor
> performance you're seeing, though.

In the sql method in question has 1 dtml-sqlvar in it. Everything else is
pre-rendered (there's a method that generates the sql methods).

> 1. If you're not using LoginManager as your user folder, and maybe even if
> you are, your SkinScripts should have proxy roles.  Proxy roles can slash
> the overhead of performing security checks, and are usually appropriate for
> SkinScripts because SkinScript is "internal" to the object and Zope
> security will apply to the results of the SkinScripts as well.

Most of my tests have been with the builtin user folder. But as I mentioned
above, proxy roles don't help much.
 
> 2. Why are you retrieving 200 dataskins?  Is this all in one transaction?
> If so, you may have a design issue.  Multi-object operations should
> generally be implemented as domain-specific methods on the Specialist, so
> they can be implementation-specific (and thereby take advantage of speedups
> like a ZCatalog index or specialized SQL queries).  In other words, you
> probably want to iterate directly over a 200-row query result from an
> SQLMethod, rather than retrieiving 200 DataSkins.  (You're accessing over
> 10 times as many DataSkins in one transaction as our most complex ZPatterns
> application touches in its most complex type of transaction, which involves
> about 4 or 5 Specialists at once!)  Pretty much, if you're displaying a
> list or report or doing some kind of summary analysis or mass update, it
> belongs in an implementation-specific method in the Specialist.

Yeah... this is mainly a design flaw. I and others at my company were
working on various parts of this project in parallel. I was working on the
base level ZPatterns model. They were working on client apps. We missed the
communication bus at some point and they got to thinking of Specialists as
DB tables. Something that needs to be addressed in the future, but it
doesn't help me now. :(
 
> 3. If you *really* need 200 dataskins in one transaction, and the overhead
> is *really* in SkinScript, then you can bypass the overhead by writing an
> AttributeProvider of your own in Python.  I'm guessing, however, that
> SkinScript per se is not the source of the slowdown, and you need to look
> more closely at the things the SkinScript is calling.

I think I'll have the developer just use a SQL method to get the data. This
is how it should be done anyways... 

Thanks for the reply.

-- 

John Eikenberry [jae@kavi.com]
______________________________________________________________
"A society that will trade a little liberty for a little order
 will deserve neither and lose both."
                                          --B. Franklin