[Zope-ZEO] Advice

[Greg Ward]

On 21 September 2000, Jim Fulton said:
> > [Jim steps in]
> > > I'd like to hear more about the problems you've had and how
> > > you overcome them in a different system.
> 
> Hm. You really haven't answered my staging question.

So I didn't -- sorry!  Here are the difficulties we had trying to
develop a non-trivial web application with Zope.  Most of this was late
1999-early 2000 (although we're still living with the code written
then).  We only used DTML Methods & Documents and External Methods;
Python Methods didn't exist when we started; Andrew experimented with
writing Products and didn't like what he found; and I don't think we
ever looked into ZClasses.  So that may have been part of our problem, I
don't know.  Anyways:

  * poor support for multiple developers: if two people are hacking
    away on the same code, they see each other's changes
    instantaneously, instead of being mediated through CVS or
    something similar.  (Versions don't help when a lot of your code
    is in external methods.)

  * hard to run development/staging/live servers, because it's hard
    to sync code.  If everyone ran a development server with their copy
    of the code, then developer isolation is achieved ... but then how
    do you integrate the code bases for testing on the staging server?
    If the code was in the filesystem, you'd use CVS and it wouldn't be
    a big deal.  With the code in a database, you need some custom tool
    that knows all about that database, which AFAIK does not exist for
    Zope's database-of-code.

  * procedural abstraction is painful, part 1: factoring out a new
    function requires creating a new object (DTML Method) to put the
    code in.  The obvious analogy when using Zope is folder=directory
    and object=file, so it's as though I had to create a separate .py
    file for every single function and method in my system.  Ouch.  I
    just did a quick "find ... | xargs grep" on the application logic
    and presentation layers of our current system (which is maybe 15-20%
    finished) and found 423 functions, methods, and templates across 61
    files.  In Zope, that would be 423 distinct {DTML,External,Python}
    Methods -- ouch!  (More likely, we would do less procedural
    abstraction, have more cut 'n pasted code, and thus fewer objects.
    That's what kept happening when we were actively developing with
    Zope.)

  * procedural abstraction is painful, part 2: procedures (DTML methods)
    don't have formal parameters, they just happen to inherit whatever
    namespace they were called from.  Yikes!  This is like Perl, only
    worse; at least with Perl, you *can* have lexically-scoped
    variables.

  * no revision control on source code.  I can't go to my Zope
    database-of-code and fetch the current state of the system on Jan
    15, 2000 because the database has probably been packed since then.
    This packing is necessary because every tiny, trivial change is
    stored, most of which are irrelevant.  I want to explicitly *say*
    which change-points are key, and I want the version on either side
    of those change-points saved for all eternity.  This is what RCS or
    CVS buy me, and I can't imagine doing serious programming without
    it.

  * editing code in a browser window is extremely painful and awkward.
    Yes, I know about the ability to ftp into the database; we
    experimented with that early in the Zope 2.1 days, got bitten
    by some instability bugs, and quickly turned that ability off.

  * DTML syntax is bizarre and awkward for experienced Python
    programmers.  (And I would hazard a guess that it's even more
    confusing to non-Python programmers: "_.None" bugs me to no end, but
    at least I kinda-sorta understand what it means.)

Our solution -- which is really Andrew Kuchling's brainchild, with major
contributions by Neil Schemenauer, some small miscellaneous hacks by me,
and some design ideas from Barry Warsaw -- is Quixote.  (What could be
more quixotic than yet another web application server?)  Quixote steals
Zope's two best ideas, namely:

  * URLs resolve to callable Python objects

  * web pages are subroutines that take an HTTP request as input and
    return an HTTP response and an HTML page (although, as with
    Zope, the HTTP response object is an input/output parameter,
    and the HTML page is the sole return value)

Come to think of it, these are really just two aspects of the same
insight.  Whatever.

We took those two ideas and then implemented them differently from Zope;
the goal was a system that makes sense to experienced Python
programmers.  Quixote is purely a programmer's system, because
developing web applications *is* programming -- I suspect that any
non-programmer who tries to develop complex web applications will either
fail or become a programmer in the process.

Where a Zope URL traverses Zope's database-of-code-objects, a Quixote
URL traverses Python's namespace hierarchy.  Eg. in the code I'm hacking
on today, the URL /q/user/login (I'm leaving of "http://" and the
hostname, of course) resolves to the callable Python object
"mems.ui.user.login".  (Our Quixote config file establishes that the
"/q/" URL maps to the Python namespace "mems.ui".)

Through a bit of aliasing in mems/ui/user/login/__init__.py, that
callable object is a function:

    def login (request, response):

which generates and/or processes a user login form.

Ultimately, this comes down to another callable Python object, which is
written in a Python dialect called PTL (Python Template Language).  A
simple example:

    template login_form (request, response):
        standard.header(title="Login to the MEMS Exchange", request, response)
        user_id = html_quote(request.form.get('user_id')) or ''
        """
        <form method="POST" action="login">
          User ID: <input type=text name=user_id size=20 value="%s"><br>
          Password: <input type=password name=password size=20><br>
          <input type=submit name=login value="Login">
        </form>
        """ % user_id
        standard.footer(request, response)

PTL is just Python with two changes:
  * "template" instead of "def", so the language is built around
    templates rather than functions
  * a template doesn't have an explicit return value; rather, the
    return value is built up by accumulating non-None results from
    executing each statement in the template, and then returning
    an object whose str() is the accumulated text

In the above example, three statements have non-None return values --
'standard_header()', the literal string, and 'standard_footer()' --
which are bundled up in the TemplateIO object that underlies things;
str() of the TemplateIO concatenates the strings.

Since this is based on the semantics of DTML as we understood them, this
probably looks pretty familiar to you.  We wanted similar semantics, but
a syntax we all know and love -- Python's.

Incidentally, the first version of PTL (my design) was more like
DTML/PHP/ASP/JSP/etc. -- HTML with bits of code interleaved.  We used it
for about a week and hated it; Neil had the brainwave to just use Python
and tweak the notion of a return value.  We've been using it that way
for a couple months now and it's just great.  
    
The way I see it, Zope is a content management system that has been
extended to serve as a web application framework.  I think a lot of
Zope's complexity stems from the "edit through the web" feature, which
seems nice for content systems that have lots of widely-distributed
authors who just write content and very little code.  Our situation,
though, is 5 or 6 programmers who just write code, and all have full
filesystem access on their development servers.  Some of that code is
HTML to handle the presentation layer of the UI, some of it is
application logic, and some of it is underlying domain objects.  Since
it's all written in Python (or PTL), the connections between these
layers are beautfully seamless.  And thanks to ZODB, the amount of
persistence code in the system is tiny (maybe 50 or 100 lines --
depending on how you count it -- out of 12,000 lines and growing).
(That's not counting database administrative scripts that we have
written or plan to write, which will be essential given the lack of a
general query language like OQL.)

Quixote was intended solely as a web application framework.  You could
probably use it to write a content management system, but that's not
what we're doing.  It's not intended for non-programmers, and it's
certainly not intended for people to write applications through the web.
But Netscape's <textarea> tag is not a very good source code editor, as
we found out when developing under Zope.
    
Hope this answers your question!  I'd better get back to work... like I
said, we have a loooong way to go...

        Greg
-- 
Greg Ward - software developer                gward@mems-exchange.org
MEMS Exchange / CNRI                           voice: +1-703-262-5376
Reston, Virginia, USA                            fax: +1-703-262-5367