[Zope3-dev] Vocabulariy issues

[Garrett Smith]

Jim Fulton wrote:
> Zope 3 has a potentially very cool framework named "vocabulary":
> 
>   
>   
>
http://svn.zope.org/Zope3/trunk/doc/schema/vocabularies.txt?view=markup
> http://svn.zope.org/Zope3/trunk/src/zope/schema/interfaces.py?view=log
> 
> Vocabularies represent sets of values.  They exist to support choice
> fields in schemas, which constrain values to be a choice from a set
> of values. 
> 
> I think we don't understand vocabularies well enough yet.  The
> evidence for this is the set of APIs surrounding vocabularies, which
> are very complex. (Usually complexity in an API reflects poor
> understanding of a problem.) 
> 
> (I'm working on a proposal for implementing PAS for Zope 3.  I need
> to work out how to provide searching capabilities for PAS plugins. 
> I've spend a lot of time trying to figure out the vocabulary
> frameworks to support this application.  I think I've managed it,
> with Gary's help.) 
> 
> Let's start with some basic issues.
> 
> 1. A vocabulary is defined as a set of terms.  Terms are objects
>     having 'value', 'token' and 'title' attributes.  The 'token'
>     attribute is a a printable ASCII string that provides an
>     attribute for a value to be used in forms.  This is clearly
>     UI-specific and shouldn't be in the underlying model.  The
>     'title' attribute is a bit suspicious as well, as it is primarily
> useful for displaying values in forms. 
> 
>     I think it would be better to simplify vocabularies to represent
>     sets of values, rather than terms.  Vocabulary views can then be
>     used to provide access to UI information, such as ids and titles.
> 
> 2. Choice fields require IVocabulary objects, which means that they
>     can only be used with vocabularies that support iteration.  Large
>     vocabularies shouldn't attempt to support iteration. (See below.)
> 
> One of the greatest benefits of vocabularies is their potential to
> support very large (possibly infinite) sets of objects.  Attempted
> support of large vocabularies is a source of major complexity in the
> current framework. This support centers around the concept of queries
> and query views.  Sadly, despite the apparent support, vocabulary
> queries and query views don't actually work with the current widgets.
> (I know they did work once, but they don't work now.)
> 
> Before discussing the query issues, let's step back and look at the
> problem that queries want to solve:
> 
> For large vocabularies it isn't practical to list the values for
> selection. Rather, we need some way to query the set of values to
> find values we might want to use.  Different vocabularies will need
> different query interfaces, depending on the mechanisms (e.g.
> database indexes) available.  Having gotten some search results, we
> can use them to update selected values. How we update the selected
> values depends on the way the vocabulary is being used.  Consider 3
> common cases: 
> 
> a. We have a simple choice field.  We need to select a single value
>     from the vocabulary.  After doing a search, we can choose to use
>     one of the serach results as the selected value.
> 
> b. We have a field that is defined as a set of choices.  In this case,
>     we need to select multiple values.  After doing a search, we might
>     choose to add some of the search results to the list of selected
>     values.  Note that the selected values will often include values
>     not in the search results.  We can also choose to remove values
>     from the list of selected values.
> 
> c. We have a field that's defined as a sequence (e.g. List or Tuple)
>     of choices.  This case is similar to the set case (b) above, but
>     with the added requirement to sort the selected values.
> 
> In all cases, we end up with a 3-part UI for a widget:
> 
> A. We have an input form for collecting search criteria.
> 
> B. We have a results display.  This part of the UI might
>     be ommitted in the absense of a search.
> 
> C. We have UI displaying selected values.  This might include
>     controls for deselecting items.
> 
> In a GUI, parts A and B above would typically be provided in a
> separate dialog.  It's worth noting that part B will depend on
> the number of values that can be selected.
> 
> Theoretically, vocabularies support large sets ov values
> through "queries".  IVocabulary defines a getQuery method for
> getting a query object for a vocabulary.  The separate query
> object is supposed provide the necessary search support.  For
> browser based UIs, an IVocabularyQueryView interface provides an
> API for delegating implemention of parts A and B described above.
> Implementation code in 'zope.app.form.bowser.itemswidgets' seems
> to support query veies, but the support is incomplete (decoy).
> 
> Now, continuing with issues related to querying:
> 
> 3. The views in itemswidgets are designed and registered for
>     IVocabulary, rather than IBaseVocabulary. In particular,
>     they rely on vocabulary iteration and create selects lists
>     listing all vocabularies.  They should, instead, maintain
>     data copied from query results when query views are used.
> 
> 4. The views in itemswidgets never call the performAction
>     method defined by IVocabularyQueryView and, therefore can't
>     information from search results.
> 
> Really, choice or choice-based widgets need to have very different
> behavior in the presense of large vocabularies.
> 
> I really think we can simplify all of this quite a bit.
> One challenge is backward compatability. There's quite a bit
> of code that has been written for the existing vocabulary APIs.
> 
> I propose to create a new vocabulary-like framwork to replace the
> current one.  For the sake of discussion, lets call these sources
> (of data from which we can choose).  I'm open to alternate name
> suggestions.
> 
> - Define the following interfaces:
> 
>    class ISource(Interface):
> 
>        def __contains__(value):
>            """Returns True if the value is available from the
> source""" 
> 
>    class IIterableSource(ISource):
> 
>        def __iter__():
>            """Iterate over the values in the source"""
> 
>        def __len__():
>            """Return the number of values in the source"""
> 
>    class ITerm(Interface):
> 
>        value = Attribute(
>            "value",
>            "The value used to represent vocabulary term in a field.")
> 
>        title = TextLine(title=_(u"Title"))
> 
>        id = ASCII(
>            title=u"Id",
>            description=u"""Unique identifier for a value
> 
>            The value of this attribute must be a non-empty 7-bit
>            string. Control characters are not allowed.
>            """)
> 
>    class ITerms(Interface):
> 
>        def getTerm(value):
>            """Return the ITerm object for the given value
> 
>            LookupError is raised if the value isn't in the source
>            """
> 
>        def getValue(id):
>            """Return a valie for a given identifier
> 
>            LookupError is raised if there isn't a value in the source.
>            """
> 
>    class ISourceQueryView(Interface):
>        """View support for querying non-iterable vocabularies
>        """
> 
>        def setName(name):
>            """Set the name used to compute the form field names
>            """
> 
>        def value(value):
>            """Update a selection value to reflect use of query results
> 
>            The value should be modified *only* if the user has
>            elected to apply serach results to the selected value.
> 
>            Return a new value, reflecting updates, if any.
>            """
> 
>        def renderInput():
>            """Return a rendering of the input portion of the
> widget.""" 
> 
>        def renderResults(value):
>            """Return a rendering of the results portion of the widget.
> 
>            `value` is the current value represented by the widget.
>            """
> 
> - When we want either a widget for a choice field or a widget for a
>    collection of choice fields, we'll use a multi-view widget on the
>    field and the field's source.
> 
> - We will use different widgets for sources and iterable sources.
> 
>    For iterable sources, the widgets will use select lists on the
>    values obtained through iteration.
> 
>    For non-iterable sources, the widgets will get source query views
>    (ISourceQueryView) and use them to implements parts A and B of the
>    UI described earlier.  The source query views will be multi-views
>    on the source and the field. The extra indirection through source
>    query views allows the implementations of UI parts A and B to be
>    separated from the implementation of part C.  The implementation
>    of part C can be fairly generic.
>
> - Choice widgets will use 'ITerms' views on the choice sources to
>    implement 'ITerms'.
> 
> - We'll support both sources and vocabularies by making
>    vocabularies specialized sources.  In fact, for iterable sources,
>    we may just reuse the existing iterable vocabulary support by
>    adapting iterable sources to iterable vocabularies (as views).
>    We'll remove the non-working (decoy) query support from the current
>    vocabulary widgets.
> 
> Thoughts?
> 
> Jim

Let me see if I have this right:

- The motivation to create a 'source' framework is to have something
that provides values rather than terms. This simplifies the 'source'
part of the framework.

- To provide titles and ids ('tokens' in vocab-speak) for values, we
have ITerms, which is a view to ISource. So we'd use it like this:

  zipcodes = zapi.getUtlity(IIterableSource, 'zipcodes')
  terms = zapi.getView(source, '', request, providing=ITerms)
  for value in zipcodes:
    term = terms.getTerm(value)
    value, term.id, term.title
    # e.g. u'60187', '60187', u'60187 (Wheaton, IL)' 

(Zipcodes would probably not be iterable, but it's just an example.)

With this approach, one could provide alternate ways (named views) of
displaying zipcode titles without redefining the source.

- The existing vocab framework doesn't really support querying
interfaces, though it sort of pretends to. For this we create
ISourceQueryView.

Questions:

- Apart from backward compatibility, is there a reason to keep the
existing vocabulary framework around? Is this something that will/should
be deprecated at some point?

- I'm having a hard time placing ISourceQueryView in context with
IWidget. It seems that ISourceQueryView is UI for a specific
IInputWidget that is displayed as a separate dialog. This is similar to,
e.g., a date/time widget that provides a separate dialog for navigating
a calendar UI. Any input widget that provides a UI that has its own
request/response support would fall into this category.

I'd like to see the widget framework enhanced for this sort of thing.
The typical scenario would look something like this:

1 - User displays an edit form containing input widgets. One of the
widgets contains a button for browsing or searching for values.

2 - User clicks the 'browse' button on the widget to display the
browse/search dialog.

3 - The current values in the edit form are saved (session data, or
embedded in the browse/search dialog).

4 - The user interacts with the browse/search dialog until she clicks an
OK button.

5 - The user is redirected back to the original edit form, where the
newly selected values (if any) are displayed along with the saved values
from step 3.

This scenario would be different if the dialog is displayed in a
separate window or an iframe (probably the more common use case for
modern UIs). E.g., step 3 would not be necessary and their would be some
browser scripting involved.

I know Gary has done a lot of work in this area with his widgets --
might this not be the time to enhance widgets/forms for this?

 --Garrett