[Zope3-dev] Proposal for file-system synchronization in Zope3

[Andy McKay]

> > I would like to make it easier for:
> > a) users to move objects between storages and products
>
> What do you mean by "storages"? "products"?

By storage I mean ZODB or file system. Its a rather overloaded term and
probably not the best one to use.

By product I mean code that is used to access the object, such as the
synchronising tool or a serving tool such as File Directory View or Local
FS.

> > To accomplish this I believe we need a common API for moving objects
between
> > the ZODB and the file system. One way to achieve this goal is to expand
the
> > import and export functionality we currently have in Zope to this new
"file
> > editable" standard.
>
> The first step is to have an API for providing a file-system
representation.
> Given that, various applications might be written to do synchronization.

Right. We're agreed there.

> > Once we have a common API and standard for moving objects and
representing
> > objects on the file system, we will have more flexible and powerful
tools
> > that work with each other.
> >
> > Example:
> > - I want to serve DTML and Python Scripts in the ZODB from the file
system.
> > How do I accomplish this?
> > - sync the objects using the Synchronization
> >  - objects pass through the common API and are written in a standard
format
> > - serve the objects using FS Directory View (or some similar product)
> >  - objects are read using the standard format and served to the user
>
> I believe strongly that synchronizing with and serving from the file
system
> are too different from each other to try to combine them.  For
synchronization,
> you need a lossless representation that is convenient for people to
manage.

Im not trying to combine them, I'm trying to make sure they can be combined
easily if people wanted to.

> For serving from the file system, you don't need a lossless representation
and you need
> a representation that's tuned for serving.

I disagree, I believe that view is too simplistic. I would divide serving in
to 2 use cases: Development and Production.

I'm not arguing that your tool should handle this, just that there are valid
use cases to consider.

In production I want speed, performance, robustness and scalability that the
ZODB and ZEO provide.

In development I want ease of use and development. Most enterprise and
professional zope development compaines I know seperate development and
production server, many developers do most of their work from a personal
sandbox. To have a lossless representation of a Python Script on the file
system that is automatically served from Zope is a powerful thing to have.

In the synchronization model you propose, I would have to sync from the ZODB
to the filesystem, edit the file in the file system. The sync back to the
ZODB. I believe there are two unnecessary steps for development. Two
examples of products that already allow you to do this? Refresh and File
Directory Views both handle automatic refreshing and updating during
development, for production this is turned off and performance is improved.

Further this brings another issue of 3-way merges and diffs. There are 3
possible places the object can be changed (in most development
environments), ZODB, CVS (or other source control) and the file system.
Eliminating one of those sources for development makes life easier for your
average developer.

> > I just want to ensure that if we check something into the core, its a
> > standard, flexible, extensible system for the good of all mankind.
Whilst
> > the syncrohnization proposal is great, I would rather see:
> > - this common API and standard file format
>
> What common API and file format?

Part of the problem is that reading the proposal I get the feeling it is
doing 2 jobs. Where as I see them as distinct. When you mention "Find out
what's changed in Zope or on the file system", "The initial tool will be a
command-line tool modelled" these suggest you are mixing together the
synchronization tool with the API. To me they are two different tasks.

I think we both agreed on this point earlier in this email when you said
"The first step is to have an API for providing a file-system
representation."

If you are going to put this in Zope 2, I would like to see an API that is
integrated into the current Zope import / export system. For example:

def manage_exportObject(self, toxml=None, format=None, ...)
  """
  toxml is a legacy parameter
  where format is xml, zexp or plain text
  """

def manage_importObject(self, replace=1, ...):
  """
  replace overwrites the objects rather than updates them
  if file is a folder, it will assume this is plain text format and look for
Entries.xml
  """

In the end I'd like to see the file format for these objects specified so
there it is a reference we can point to.

> The proposal includes both. Do you have any comments
> on them?

Yes they look great. I was glad to see you noted the xml "fairly friendly to
tools like diff", that is important.

Can I just check if I wanted to make an object from the file system using
your example, say a Python Script called "foo.py", I would have to:
    - create my python script in the /tmp/folder dir
    - add it to the entries.xml file in the /tmp/folder/@@Zope dir
    - edit the contentType in /tmp/folder/@@Zope/Extra/foo.py

Could this be made any easier? If the file has a recognised extension such
as .py, could this be figured out? What does entries.xml contain, does this
contain the contentType for example?

Is there any need to handle data that has been altered incorrectly on the
file system? Such as give an object a property it shouldn't have or have
access to?
--
  Andy McKay
  www.agmweb.ca