[Zope-dev] (LONG) RE: [SmartObjects] Wrap-up of the discussion going on on zope-dev?

Albert Langer Albert.Langer@Directory-Designs.org
Tue, 15 May 2001 06:33:28 +1000 

[Joachim Werner]
As most of you might have recognized, there is a very active thread
(actually two interwoven ones) about RO-mapping etc. going on on zope-dev.
I'd be very happy if someone could wrap up the stuff from there on the
SmartObjects list. ;-)

[Albert]
Thanks for the pointer. As I read them all through at once, after
seeing your pointer, I made notes before joining the discussion,
which may help, though far from a "wrap-up".

* SUMMARY *

There seem to be 6 separate topics:

1) Nature of SmartObjects, ZPatterns and Transwarp frameworks and
relation to Zope/ZODB framework.

2) Improvements to ZCatalog search efficiency

3) Query syntax - see Zwiki:
http://dev.zope.org/Wikis/DevSite/Proposals/UnionAndIntersectionOperations

4) Schema transparency (an oxymoron?)

5) Storage of ZODB objects in separate DBMS tables per class - "Object
Mapping".

6) Requirements for "Object Relational Mapping".


* DETAILED NOTES *

Here's a list of all the items I found in the May archives
with the term "ORMapping" in the subject line, from thread
"Experiments in ORMapping" and interwoven thread "oodb
philosophics was [above]" until I subscribed. Will comment
later on subsequent messages rather than making notes.

If there are any other subject headings missed, or other postings
(including earlier months), please let me know.

Highlights before links attempt to be objective but are limited
by space, by my specific interests in the topic and by my lack of
thorough understanding of Zope internals and consequent previous
non-participation in [zope-dev] with consequent lack of appreciation
of where various posters are "coming from".

Quotes following links are verbatim extracts (not necessarily
in same order or in context). "Must read" means important message
with no or inadequate highlights or extracts given here.

I have omitted numerous affirmations that ZODB meets most current
Zope requirements (perhaps with improvements to ZCatalog)
in view of the fact that nobody appeared to be arguing otherwise.

BTW the date order used in list archives could be better in numeric
order as that might more closely reflect which messages are likely
to have been seen before replying. I have re-arranged.

Starting from 10 May 2001:

Shane Hathaway (DC) - start of discussion - must read
http://lists.zope.org/pipermail/zope-dev/2001-May/011101.html
includes link http://www.zope.org/Members/hathawsh/ormapping.tar.gz
overview in file: sketch

Tino Wildenhain - asks about ZPatterns, ZClasses, doesn't want
"purely relational application server".
http://lists.zope.org/pipermail/zope-dev/2001-May/011102.html

SH - answers TW, says not hard to implement by mapping classes
to a table.
http://lists.zope.org/pipermail/zope-dev/2001-May/011103.html

TW - answers SH, prefers improved OODB because doesn't like
mapping classes to schema.
http://lists.zope.org/pipermail/zope-dev/2001-May/011105.html

SH - answers TW, enhanced ZODB storage (RDBMS and BerkelyDBM) is in some
ways the best OODB.
http://lists.zope.org/pipermail/zope-dev/2001-May/011107.html
"The other motivations for an RDBMS are (1) people have existing schemas
and want Zope to access the same data as their existing apps, and they
want it to be transparent, and (2) tables with millions of entries are
easily stored in Zope but the perception is that the catalog isn't as
fast as a database index.  No one has done any tests AFAIK." [...]
"That's one reason ZODB is so nice.  You can write an application without
writing a formal schema."

Casey Duncan (Kaivo) - suggests Matisse or Objectivity but limited by
not supporting ZODB versions. BerkelyDBM best soon. Mentions slow XML
storage startup.
http://lists.zope.org/pipermail/zope-dev/2001-May/011108.html

JW - already 2 OR mapping projects, SmartObjects and TransWarp,
no duplication because Phillip from Transwarp participating in
SmartObjects list.
http://lists.zope.org/pipermail/zope-dev/2001-May/011109.html

TW - answers CD, XML just another pickle format.
http://lists.zope.org/pipermail/zope-dev/2001-May/011110.html

SH - answers JW. SmartObjects, ZPatterns and Transwarp require new
database API instead of maintaining transparency like ZODB. Projects
should look at replacing parts of ZODB instead of adding complexity.
http://lists.zope.org/pipermail/zope-dev/2001-May/011111.html
"ZODB has pieces that can be split apart and replaced as needed, such
as caching, persistence, transactions, the "pickle jar", the
multi-threaded connection factory, and the storage layer.  I'm hoping
we can achieve OR mapping by only replacing the "pickle jar", i.e.
Connection.py."

Cees de Groot - answers SH. Confirms advantage of not having to
write formal schema, migrating from PostgreSQL to ZODB for that.
File Storage faster than Oracle and is basic transaction log
so nothing could be more reliable.
http://lists.zope.org/pipermail/zope-dev/2001-May/011124.html
"Are people using ZODB for non-Zope data? I'd be very interested to discuss
things like emulating extents, patterns for indexing, etcetera..."

Chris Withers
http://lists.zope.org/pipermail/zope-dev/2001-May/011125.html
"doing this transparently would be unbelievably cool :-)
So, if I had a.n.other SQL table containing some attributes,
how would you see that mappign to Zope objects?
What meta_type would they be?
Where would they show up?"

JW - answers SH, SmartObjects (SO) summary, must read.
SO is framework, not just adding OR. Design assumes cannot
change Zope core but would prefer if Zope extended to handle
query and storage. Wants more efficient "cross-tree" queries
than ZCatalog (or improved ZCatalog). SO is also "better
ZClasses". Transparent for users but needs ZMI and DA plug-in
as products.
http://lists.zope.org/pipermail/zope-dev/2001-May/011129.html

JW - answers CW, SO needs non-transparent API to define mapping.
http://lists.zope.org/pipermail/zope-dev/2001-May/011130.html

JW - answers SH, SO has something similar to ZClasses that
defines the schemas to create both the classes and the tables
on the fly.
http://lists.zope.org/pipermail/zope-dev/2001-May/011131.html

JW - speed tests should be done on ZODB. If could store full
text of 30GB of Word and .pdf with ZCatalog indexes, and
complete zope.org list archives efficiently would use for all
document management needs. Also java ODBMS requires recompile
for changes but Zope [ie python, not just ZODB - AL] can just
refresh. ZODB has longer success record than other ODBMS.
http://lists.zope.org/pipermail/zope-dev/2001-May/011134.html

SH - answers JW. Adapting ZODB for OR can just replace
Connection class in custom_zodb.py - no change to Zope core.
ZCatalog not as flexible as an SQL query but can efficiently
do filters on one table at a time. *Alternate* ZCatalog would
be useful. CMF can restrict results based on user permissions.
http://lists.zope.org/pipermail/zope-dev/2001-May/011146.html
"Here's the goal I envision for OR mapping: to be able to move between
OODBMS and RDBMS seamlessly.  It makes sense to develop on top of ZODB
then move to an RDBMS for testing and deployment."
"It sounds like [SO] are tackling a lot of things at once.  You might
consider dividing into separate projects."
"You see, I think it is not necessary to create a programming framework
if the goal is OR mapping.  The framework is already defined, and it's
Python / ZODB.  But SmartObjects seems to have many loosely related
goals, making it difficult to assist."

Ken Manheimer (DC) - answers JW. Speed tests done for mailing
lists. Searches fast, loading slow. ZODB as flexible and
powerful as he could want for python Zope programming. May
have been canned searches, load testing may have resulted
in improved tuning since. Eventually would like to incrementally
stuff messages as they arrive. No experience with other ODBMS,
very little with RDBMS.
http://lists.zope.org/pipermail/zope-dev/2001-May/011147.html

CD - answers CdG. Need better general query language than
ZCatalog like those for most OODBMS and all RDBMS.
http://lists.zope.org/pipermail/zope-dev/2001-May/011151.html

David Brown - answers CD. Transparent access to and manipulation of
data in python without having to think in SQL is more important
than generic query language that can access different data sources,
especially in view of SQL implementation differences.
http://lists.zope.org/pipermail/zope-dev/2001-May/011152.html

JW - answers KM. Fast search results promising. What exactly
is problem for loading? Anything likely to be slow on large load.
http://lists.zope.org/pipermail/zope-dev/2001-May/011153.html

JW - answers DB. SQL can query objects in more than one container
with an association table. No good way to implement or efficiently
query "many to many" relations in object hierarchies.
http://lists.zope.org/pipermail/zope-dev/2001-May/011154.html

JW - answers SH. SO still collecting "feature requests". May
need separate projects. SO not good name for most features of SO.
Filtering on one table at a time inadequate for ad hoc data mining
across many object relations.
http://lists.zope.org/pipermail/zope-dev/2001-May/011155.html
"But "Python/ZODB" is not a high-level framework. Like Zope is not the CMF.
Nobody would implement GUIs in pure C now. They are using tools for that.
And we don't want to have to implement things like HTML forms, event
handling, etc. over and over again. We want to just configure and "compile".
Of course this will be within the Python/ZODB framework, i.e. all the
additional functionality will be "support classes" or mix-in classes like
the ones Zope already offers (e.g. CatalogAware)."

CD - answers DB. Python great for OO manipulation of data but not for
generic queries. Python query language - needed eg criteria expression that
returns objects meeting the criteria. ZCatalog argument list too limited.
http://lists.zope.org/pipermail/zope-dev/2001-May/011157.html
Reference to following link - must read.
http://dev.zope.org/Wikis/DevSite/Proposals/UnionAndIntersectionOperations

CdG - answers DB. Has Python objects above homebrew O-R layer, very useful.
Should be possible to express queries as python code with mapping layer
generating equivalent SQL from parse tree (like SmallTalk lambdas).
http://lists.zope.org/pipermail/zope-dev/2001-May/011158.html

DB - answers JW. Python can map specific indexes and mappings more
easily and efficiently without re-writing database code or forcing
developers to think in SQL.
http://lists.zope.org/pipermail/zope-dev/2001-May/011159.html

CD - answers CdG. Must be safe for "through the web" scripting.
http://lists.zope.org/pipermail/zope-dev/2001-May/011160.html

CdG - answers CD.
http://lists.zope.org/pipermail/zope-dev/2001-May/011162.html
"I think that TTW scripting and heavy duty application development are very
incompatible with each other, so that's not a problem :-)"

Phillip Eby - answers SH, JW. Explanation of Transwarp, must read.
http://lists.zope.org/pipermail/zope-dev/2001-May/011163.html
"I'm not quite clear on how exactly you suggest mapping from RDMBS ->
ZODB.  There's a *significant* (IMHO) impedance mismatch between ZODB's
arbitrarily identified variably structured single records and SQL's
content-identified fixed-structure record sets.  This is what application
frameworks/toolkits (such as your own DBAPI product) are needed for."
"[Develop on ZODB then move to RDBMS requirement] can *only* be met through
a
common API or access pattern/framework/what-have-you, be it DBAPI,
ZPatterns, SmartObjects, or TransWarp.  The ZODB is both "too powerful" (in
its flexibility) and "too weak" (in lack of any ZODB-level notions of
record sets, schemas, and indexing) to be useful as a
cross-database-platform API.[...]"
"Anyway, all I'm really trying to say is, if you want implementation
independence, you have to have implementation hiding.  ZODB makes it easy
to store Python objects with hardly a second thought.  However, if you plan
to use something other than ZODB for *any* part of your object at any time
in future, you need an API that hides the implementation."

SH - answers PE. Suggestion was to implement ZODB DB interface with
different Connection.py.
http://lists.zope.org/pipermail/zope-dev/2001-May/011164.html
"If you implement this at the Storage level, yes, there is a major
mismatch.  But at the Connection level it makes a lot of sense.
Connection basically exposes a pile of pickles as objects; an OR mapping
exposes a complex schema as objects.
I think that understanding will change the rest of your response. :-)"

CdG - answers JW.
http://lists.zope.org/pipermail/zope-dev/2001-May/011165.html
"[...]AFAIK ZODB and most OODB's
store an object only once, keyed by its object id. The rest is just
references
through that oid, so objects that belong to more than one container can be
added to all these containers and n:m relations are implemented by having a
list of objects on both sides."

CdG answers PE. Can trigger external changes from application
code without major risk or mess - gives example.
http://lists.zope.org/pipermail/zope-dev/2001-May/011166.html

CD - answers CdG. Acquisition may cause problems for n:m relations.
http://lists.zope.org/pipermail/zope-dev/2001-May/011167.html

CdG - answers CD. Doesn't want to use acquisition for business
objects anyway.
http://lists.zope.org/pipermail/zope-dev/2001-May/011168.html

PE - answers SH.
http://lists.zope.org/pipermail/zope-dev/2001-May/011171.html
"Nope.  As far as I can see, all that does is leverage ZODB Connections as a
crude sort of Rack that only provides a mapping-like interface for
retrieving objects, without helping any of the higher-order needs of an
application.  I guess if you define O-R mapping as "can you store and
retrieve an object's properties from a database", then I guess you've
succeeded at that point.  But if that was all my O-R apps needed, there
would've been no reason to use the RDBMS; I could've just used ZODB and a
BTree with less bother."

JW - Acquisition cool but sometimes sucks.
http://lists.zope.org/pipermail/zope-dev/2001-May/011180.html

JW - answers CdG. Use Symlinks (ref below) - should be in Zope.
http://lists.zope.org/pipermail/zope-dev/2001-May/011181.html
http://www.zope.org/Members/hathawsh/Symlink/index_html

(13 May no further notes - thread still picking up steam, far too
early to "wrap-up")

* ELSEWHERE *

Another relevant posting I noticed recently was from Nils Kassube
in [zope], so I am CCing this to Nils as he seems not to be subscribed
to the relevant lists:

http://imail.iuveno-net.de/mailman/listinfo/smartobjects
http://lists.zope.org/mailman/listinfo/zope-dev

"According to the Zope Developer's Guide our favourite web application
server prevents dirty reads with his transaction support. Is it possible
to prevent non-repeatable reads and phantom reads, too? Or should I use
a more appropriate tool like PostgreSQL with support for MVCC? Is there
more documentation about Zope's transaction support? (Or is it another
case for "Read the source, Luke"?)"