[Zope-dev] DA Roadmap

Christopher Petrilli petrilli@digicool.com
Thu, 29 Jul 1999 12:33:56 -0400


Gang,

Below is the Database Adapter roadmap that I've just put up on the web site.
It covers both where we are, how we measure where we are, and where we're
going.  Some of you might find the section dealing with the future of ODBC
drivers interesting.

Please let me know directly any feedback.

Chris

Roadmap for Zope Database Adapters


  Overview

    The past few weeks, having brought the Zope 2 release to a beta level,
    have brought to the attention of everyone that the current design and
    architecture of various DAs is incompatible with use in a fully
    threaded environment.  This document hopefully will clarify some of
    what the various level of compatibility is, as well as comments on
    individual DAs, and their future.


  DA Thread Safety Levels

    In order to quantify in a uniform manner the compatibility of various
    DAs with Zope2, we have developed the following "levels" for database
    adapters to implement.


    Level 1

      All database adapters which implement this level of compatibility
      must run in a single thread as they have no level of thread safety.
      This currently applies to nearly all DAs.  This means in ZServer that
      Zope runs in a single thread.


    Level 2

      Level 2 database adapters must be safe to use in a multi-threaded
      Zope instance.  This can take multiple behavior patterns:

        o Internal locking/mutexes for single DA access

        o Global locking through the transaction manager if appropriate

      While a sub-optimal solution, it should be usable on any application
      that has minimal, or infrequent, database access.  This means that
      all access patterns will be serialized.

      In addition, all DAs at this level *must* release the Python global
      interpreter lock.


    Level 3

      A database adapter implementing Level 3 compliance must be fully
      safe running in a multi-threaded Zope instance.  In addition, it
      must support:

        o Multiple connections in the same thread

        o Multiple connections in different threads running simultaneously

      Note that there are instances (see the CLI information below) under
      which the DA might implement Level 3, but the underlying client
      libraries are somewhat different.


  Transaction Manager Issues

    Zope has the concept of a Transaction Manager which deals with the
    integration of Zope transactions and transactions in an external data
    source.

    New Semantics

      With the introduction of Zope 2, there are new Transaction Manager
      semantics, described as below.  If you wish to bring a DA into line
      with the new specification, these methods are what you must
      implement.

        o 'self._begin' - This is called at the beginning of every
          transaction.  Depending on your database, and its implementation
          of "implicit transactions", you may or may not have to do
          anything here besides a 'pass'.

        o 'self._finish' - This is called when a transaction is finished,
          or committed.

        o 'self._abort' - Obviously this is called when the transaction
          involved has been aborted.  It should *always* roll back the
          transaction in the database.


    Integration Requirements

      Not all DAs talk to data stores which have transactional semantics
      These Database Adapters should not implement integration with the
      Transaction Manager unless it makes complete sense.  However, all DAs
      which deal with data stores which do have a transactional semantics
      should integrate completely with the Transaction Manager.


  Relational Database Adapters Available


    OracleDA (v2.0.0)

      The current Oracle DA is based on the 'dcoracle' package, and is not
      currently thread safe.  It operates at L1 currently, although L2 is
      trivial.

      *Near term progress.* Before the final release of Zope 2.0, the
      Oracle DA will be renamed to ZOracleDA, and will be brought up to
      Level 3 compliance.

      *Long term progress.* Plans have been made to split the ZOracleDA
      into two seperate DAs:

        o ZOracle7DA - Compatible with Oracle 7 OCI interfaces.

        o ZOracle8DA - Comaptible with Oracle 8 OCI interfaces.  This will
          also introduce some level of support for LOBs.

      *Support availability.* Commercial support is available.  Note that
      this is support for the DA, Digital Creations will not be selling
      general support for 'dcoracle'.


    ZSybaseDA (v2.0.0)

      The current internal Digital Creations implementation of a Sybase DA
      (using the 'ctsybase' Python level interface) is operating at Level
      1, with some work having been made at making it Level 2.  Currently
      it has a Sybase-level memory leak when at Level 2.

      *Near term progress.*  Before the final release of Zope 2.0, the
      ZSybaseDA will be brought up to Level 2 DA.  This is being forced by
      customer requirements.

      *Long term progress.* While we desire to bring the ZSybaseDA up to
      full Level 3 compiance exists, it will not happen until customer
      requirements have driven it.

      *Support availability.*  Commercial support is available.


    MySQL Database Adapter (v1.1.3)

      The current MySQL adapter is at least Level 1 compliant, however, it
      does not currently implement the new transaction manager symantics.

      *Near term progress.*  We would very much like to transfer ownership
      of the MySQL DA to someone who has a desire to keep it maintained.

      *Long term progress.*  None.

      *Support availability.*  None.


    SOLID Database Adapter (v2.0.0)

      Due to the recent change in SOLID Technologies marketing direction
      and a clarification from their staff that they are no longer
      interested in the "stand alone" database market, we have deprecated
      support for this module.

      *Near term progress.*  None.

      *Long term progress.* We think if desired, this could be implemented
      in the upcoming replacement rather than in a seperate module.

      *Support availability.*  None.


    ODBC Database Adatper (v3.0.3)

      The current ODBC DA is not compliant with any level of Zope 2
      Database Adapter thread safety.  In addition, it is not integrated
      with the new transaction amanger symantics.  This is a serious
      problem.

      *Near term progress.* By the final release of Zope 2.0, the ODBC DA
      will be modified to be at a minimum Level 2 compliant, based on the
      available "thunk" mechanism as well as transaction manager
      integration.

      *Long term progress.* See information below on the future of a X/Open
      CLI interface DA.

      *Support availability.* Commercial support is available.


  SQL/CLI Database Adapter

    In the process of reviewing the desirability of an IBM DB2 database
    adapter, it has come to our attention that both ODBC and DB2 are
    derived from the same X/Open and ISO specification called the "Call
    Level Interface", and are both compatible at that level.  Because of
    this, we will be transitioning to a new design for the ODBC DA, as well
    as integrating support for IBM DB2 into the same DA.


    Background on SQL/CLI

      The X/Open Company and the SQL Access Group jointly defined a
      specification for a callable SQL interface reffered to as the *X/Open
      Call Level Interface*.  The goal of this interface is to increase the
      portability of applications by enabling them to become independent of
      any one database vendor's programming interface.  Most of the X/Open
      Call Level Interface specification has been accepted as part of the
      ISO Call Level Interface International Specification (ISO/IEF
      9075-3:1995 SQL/CLI).

      Microsoft developed a callable SQL interface called Open Database
      Connectivity (ODBC) for Microsoft operating systems based on a
      preliminary draft of the X/Open CLI. Version 3 of ODBC conforms to
      almost all of the ISO SQL/CLI. ODBC 3.0 does contain considerable
      functionality that is not part of the International Standard; much of
      this is being added to the next draft of that standard.


    A Merged DA

      This new DA will be called the CLIDAzope :-) Actually, we've not come
      up with a name, but it will be based purely on the CLI specification
      intially, with small subclasses to deal with ODBC and DB2
      individually.  This will allow us to leverage two seperate
      environments at the same time.  Eventually we will integrate with
      DB2s excellent (Oracle are you listening?  :-) LOB support, and work
      to integrate with the support in Microsoft SQLServer 7 wherever
      possible.


    Relation to UNIX Platforms

      Many people have asked about ODBC support on UNIX, and this is the
      solution.  What this provides is the ability to link directly with an
      SQL/CLI compatible library without having to deal with the issues of
      an ODBC Driver Manager.  In addition, we hope that this will also
      reduce the performance impact of the driver manager.


  Non-Relational Database Adapters

    In addition to integration with relational databases, there are also
    various DAs which do not talk to "relational" data stores.  These
    include:

      o LDAP

      o Ultraseek

      o IMAP

    These databases are covered by the same thread-safety levels, but may
    or may not be integrated with the transaction manager depending on the
    paradigm that they are constructed inside.


  Deprecated Database Adapter Issues

    As you will note, we have deprecated several database adapters which we
    have handled the distribution of, and therefore they are now
    orphaned.  In order to help people who currently depend on them, we
    are offering them to the community to take up the torch of support.
    Toward that end we are willing to:

      o Transfer code ownership to a new "champion," so long as the license
        remains Open Source.

      o Provide CVS repository space for the management of the project.

    Please let us know if you have the time and desire to take up the
    banner of these DAs.


  The Kludge of the Century

    While we understand that it is desirable to have everything run
    beautifiully in a fully threaded environment, it is also unrealistic to
    assume that this will all be done within the next 48 hours, and so we
    have provided a klduge to get around this issue.  This does not escape
    the requirement for TM integration, it simply fixes the problem with
    threading in DAs that don't support threading yet, but do use the
    Transaction Manager.

    In order to bring your DA into line, you will need to subclass your
    'DB' class as follows::

      import Shared.DB.ZRDB.THUNK

      class DB(Shared.DB.ZRDB.THUNK):

      ...

    The result of this "thunk" is to create a DA that is single-threaded
    through the transaction manager.  More so, *all* DAs which are
    "thunked" are single threaded with each other.  This means that a MySQL
    DA will wait on a Sybase DA, and vice versa.  While this is
    sub-optimal, it is at least functional, which is more than many DAs
    currently can claim under Zope 2.


  Support Issues

    Where support has been discussed, please understand that we will try
    and provide support on an as available basis to all comers, however we
    can not guarantee support response times and effectiveness to any
    non-paying customers.  This should be taken into account when
    selecting a DA.

    When commercial support is offered it is offered in multiple fashions:

      o Time & materials

      o Per-incident basis

      o Yearly contract

    Please contact Digital Creations for more information about this
    option.

--
| Christopher Petrilli        Python Powered        Digital Creations, Inc.
| petrilli@digicool.com                             http://www.digicool.com