[Zope3-dev] Re: RFC: File-system synchronization proposal: ExternalFile?

[Jim Fulton]

Craeg K Strong wrote:
> 

...

> I do, however, believe that there will be overlap between the
> pluggable persistence adaptor and the synchronization stuff.  Or
> at least that you will want to use them together.

I don't think so, but I should explain a little bit why.
The file-system representation takes a file-centric hierarchical view
of the world.  The natural boundaries in this world are files and
directories.

A persistent object system has a very different view of the world.
Boundaries between objects may be much more fined-grained and may
only have a loose connection with a file view.

For example, a large image is persistent as several persistent objects,
but in a file-system representation, it should appear as one. Similarly, a
folder may use a number of persistent objects in it's implementation that
don't appear in the file-system representation.


> In an ideal world, they would both be pluggable and I would
> simply use them "together" so that I could influence the representation
> of Zope objects in the filesystem and still get automatic
> synchronization.

Note that the kind of synchronization I'm talking is not as automatic
as you might think. Synchromization is of the CVS or subversion style.
It only happens when you explicitly ask it to and it may require human
intervention to succeed (to resolve conflicts).

> The forces are tough to balance:
> a) loss-less file system representation
> b) ability to use standard file-system based tools
> c) round-trip synchronization (changes in one representation get 
> propagated to the other)

See above.

...

> We have experimented with keeping the metadata as part of the file.

...

> As well as keeping it separate.  The problem is that some metadata
> feels like part of the content of the document (like the title) but other
> metadata really feels more separate (like the list of editors, edit
> dates, comments, etc.).  One man's metadata is another man's data.

It's important to understand this distinction, even though it is
sometimes a judgement call. Data is part of the content schema.
The content objects "know" about their data. Meta-data is outside the
object schema. It is managed by other components without the content
object's cooperation.

Zope 3 makes the distinction much sharper. Meta-data are stored with a system-provided
mechanism: annotations.

> For example, I could store a single Zope object in two separate files,
> one for "data" and one for "metadata." 

That is what is proposed in the subject proposal.  Note that in Zope 3, there
are many kinds of meta-data (e.g. Dublin Core data, security assertions, etc.).
Each kind of meta-data is typ;icaly stored in a separate annotation, with a
separate file (or directory) per annotation in the proposed file-system
representation.

 > That way the "data" file
> could be structured in a way that was natural and easy for humans
> to read and tools to process.  I can always combine them easily with
> XSLT. 

Or, as the proposal describes, the data are knit back together as part
of the synchromization process.

 > The name of the metadata file could be constructed heuristically
> from the name of the data file (which corresponds to the zope ID...)

Right. See the proposal.

> In any event, I don't see any _one_ answer to this problem.  That is why I
> would like to see flexibility in how objects are represented.

Well, given the APIs, once could pretty simply adapt the provided synchronization
tool to provide different layout rules. For example, nameing conventions could be
used to provide a shallower (but wider) file-system layout.


> That way the default representation could be something convenient
> for ZServer, but those of us who use lots of non-Zope filesystem-oriented
> tools could plug in our own special schemas as needed.

The whole point of the proposal is to provide a representation that is convenient
for these tools, as opposed to a representation that is convenient for the Zope
application.  OTOH, a persistence system's job is tp provide a data representation
that's useful for direct use by the application.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (888) 344-4332            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org