[Zope3-dev] RFC: Tutorial Generator

[Stephan Richter]

Hi everyone,

now that we have testbrowser, there is a totally new range of possibilities. 
One opportunity I would like to investigate is the ability to generate 
moview-like and/or interactive tutorials based on testbrowser doctests.  I 
have written up some thoughts and I would greatly appreciate it, if you could 
comment. Here you go.

Thanks,
Stephan

--------------------------------------------------------------------------------------------------------
===================
Generated Tutorials
===================

We have recently started to write our functional doctests using the
``zope.testbrowser`` package. The advantage of using the testbrowser is, of
course, that we are describing the real interaction of the user with the
browser. This does not only provide much nicer documentation, but also
provides us with additional information about the usage of the Web
application.

Let's imagine we could use those documentation files to create fully generated
tutorials. The best about this is that I believe it is possible! This document
is meant to outline possible technical approaches to implement this tutorial
generator.

So far I have identified three desirable use cases:

(1) Generate a Flash movie that runs through the tutorial. The flash movie is
    generated offline and then is simply distributed as a static Web page/SWF
    file.

(2) Provide a live demonstration, which can be started using a simple command,
    like this::

      [Z3]$ showTutorial.py 
src/zope/app/authentication/browser/groupfolder.txt

    When this script is called, Zope 3 and a browser should be started, and
    the tutorial should start running.

(3) Provide a hands-on tutorial, where we ask the user to enter all the
    input. This could be started using a similar command to the one above::

      [Z3]$ doTutorial.py src/zope/app/authentication/browser/groupfolder.txt

The exciting part about this is that the same technical solutions could easily
be used to fulfill all three use cases. In the sections below I am going to
outline some of the technical aspects that need to be solved and suggest
solutions based on some initial research I conducted.


Reading and Evaluating the Doctest File
---------------------------------------

The code in the ``doctest`` module is quiete modular; at least it should be no
problem to parse the doctest into its documentation and example parts:

  >>> import doctest
  >>> text = open('./zope/testbrowser/README.txt', 'r').read()
  >>> parser = doctest.DocTestParser()
  >>> result = parser.parse(text)
  >>> print result[0]
  ================
  The Test Browser
  ================

  The ``zope.testbrowser`` module exposes a ``Browser`` class that
  simulates a web browser similar to Mozilla Firefox or IE.

  >>> result[1]
  <doctest.Example instance at 0xb7a2d5cc>
  >>> print result[1].source
  from zope.testbrowser import Browser

  >>> print result[1].want

Running the examples is a little bit more tricky, since the execution code is
designed only to work with a `DocTest` instance. However, I believe by writing
a common `DocTest` class, we can have better control over the runtime of the
examples; on the other hand, we might just be able to recode it.


Pushing Examples to the Browser
-------------------------------

Once we have collected the examples, they are executed using a custom
testbrowser class. The implementation of the custom testbrowser is probably
the most tricky part of the entire solution stack, because it is the piece
that communicates with the browser.

What we really want to be able to do is to push the next action to the
browser, and the browser uses some JavaScript to execute the action. On the
other hand, we want to pull information from the browser, such as the document
source, so that the comparison examples do not fail and provide the tutorial
controller with feedback about the current browser state.

Here are some approaches that could be taken:

(1) The only non-hacky solution, in my opinion, is to interact with the
    browser on the desktop and not the Web-level. For example, using KDE, we
    could utilize DCOP that allows us enough control over Konqueror to
    effectively do all necessary manipulations. Here is a small CLI session
    that I did::

      $ dcopstart konqueror
      konqueror-1000

      $ dcop konqueror-1000 | grep "html-widget"
      html-widget1

      $ dcop konqueror-1000 konqueror-mainwindow#1 openURL 
"http://localhost:7080/"

      $ dcop konqueror-1000 html-widget2 evalJS 
"tags=document.getElementsByTagName('a'); for (i=0;i<=tags.length;i++){if 
(tags[i].text.indexOf('Persons') != -1) alert(tags[i].text);}"

    The advantages of this solution are as obvious as the disadvantages. One
    of the big plusses is that we have full control over the browser process,
    even on a desktop level. Some of the disadvantages include the disability
    to run over the network and being KDE specific.

(2) The second approach basically uses the classical Web pull model. The
    actions are then evaluated using some JavaScript. We could probably reuse
    some of the Selenium code or JSON to execute the actions.

    I think there are two viable approaches for the pull:

    (a) Have a hidden frame that checks perodically for new actions.

    (b) Use a long-running stream. Uwe Oestermeier has done some prototyping
        in this area using twisted. You basically keep a stream open and push
        information through the stream as the data is available.

    The disadvantage of this approach is that it is a pretty complex
    solution. The advantage is that pretty much any browser that supports the
    `XmlHttpConnection` class will work with this solution and it can be done
    over the network.

I am still not clear on many of the technical details, especially since I do
not know JavaScript very well. Any comments and suggestions are very welcome.


Communicating the Documentation Text
------------------------------------

The demonstration would be pretty useless without providing the user with the
documentation text one way or another. There are two approaches that can be
taken:

(1) Display the text in a floating div or separate frame. This should be
    fairly easy in comparison to the callenges above.

(2) Use a free TTS tool. I have installed festival 1.95 (1.43 did not work for
    me) and it works like a charm. This would only work on local installations
    of course.

I don't think this part should pose any difficulties, whatever route we take.


The Final Output
----------------

When recording an SWF file, some more processing must occur. Benji pointed me
yesterday to `vnc2swf`. The new version is based on Python and is called
`pyvnc2swf`. It supports both, screen capturing and sound recording. I think
it should be pretty straight forward to setup the software to record the
sessions properly.