[Zope3-dev] Proposal for file-system synchronization in Zope3

[Jim Fulton]

Andy McKay wrote:
>>>I would like to make it easier for:
>>>a) users to move objects between storages and products
>>>
>>What do you mean by "storages"? "products"?
>>
> 
> By storage I mean ZODB or file system. Its a rather overloaded term and
> probably not the best one to use.
> 
> By product I mean code that is used to access the object, such as the
> synchronising tool or a serving tool such as File Directory View or Local
> FS.

OK, then I have trouble understanding the use of the term "move". Maybe
we should just drop it. ;)

....

> In production I want speed, performance, robustness and scalability that the
> ZODB and ZEO provide.

OK, good.

> In development I want ease of use and development. Most enterprise and
> professional zope development compaines I know seperate development and
> production server, many developers do most of their work from a personal
> sandbox. To have a lossless representation of a Python Script on the file
> system that is automatically served from Zope is a powerful thing to have.

I'm not convinced that "automatically served from Zope" is a good thing.

Certainly, it would be good if they were "easily" served from Zope.

I think you bring up an important point. I also like the distinction between
development and production. If we contemplate automatic update of the database
(or the running development process), it should only be for purposes of
development and need not be particularly speedy.

> In the synchronization model you propose, I would have to sync from the ZODB
> to the filesystem, edit the file in the file system. The sync back to the
> ZODB.

Yes.

 > I believe there are two unnecessary steps for development.

I think you are really only argusing that the explicit sync back to the database
is unnecessary.

 > Two
> examples of products that already allow you to do this? Refresh and File
> Directory Views both handle automatic refreshing and updating during
> development, for production this is turned off and performance is improved.

The big win here is not having to restart Zope. Another win is that refreshing
the products doesn't affect the database.


> Further this brings another issue of 3-way merges and diffs. There are 3
> possible places the object can be changed (in most development
> environments), ZODB, CVS (or other source control) and the file system.
> Eliminating one of those sources for development makes life easier for your
> average developer.

I'm not sure what you're point is. If you want to make changes in multiple
places, you have to merge. If you only make them in one, you never have to merge.
Adding the flexibility of making changes through the web doesn't really
change this.


...
> Part of the problem is that reading the proposal I get the feeling it is
> doing 2 jobs. Where as I see them as distinct. When you mention "Find out
> what's changed in Zope or on the file system", "The initial tool will be a
> command-line tool modelled" these suggest you are mixing together the
> synchronization tool with the API. To me they are two different tasks.

Right, to get the application done, I need two things:

- The application

- The API for extending the application for new object types.

It happens that the API also allows for some other applications
and different implementations of the application.

...

> If you are going to put this in Zope 2, I would like to see an API that is
> integrated into the current Zope import / export system. For example:

...

Blech. I don't think it fits all that well into the existing
export/import, but I'd be happy to punt on that if we ever do add
this to Zope 2.

...

> In the end I'd like to see the file format for these objects specified so
> there it is a reference we can point to.

Well, we can provide some formats for some common types, but the whole
point is to make sure that this is readily extensible.


> 
>>The proposal includes both. Do you have any comments
>>on them?
>>
> 
> Yes they look great. I was glad to see you noted the xml "fairly friendly to
> tools like diff", that is important.
> 
> Can I just check if I wanted to make an object from the file system using
> your example, say a Python Script called "foo.py", I would have to:
>     - create my python script in the /tmp/folder dir
>     - add it to the entries.xml file in the /tmp/folder/@@Zope dir
>     - edit the contentType in /tmp/folder/@@Zope/Extra/foo.py
> 
> Could this be made any easier?

Maybe. It's definately worth considering.

 > If the file has a recognised extension such
> as .py, could this be figured out?

Perhaps.

 > What does entries.xml contain, does this
> contain the contentType for example?

It corresponds to the CVS and subversion entries files.  It includes
a type identifier and a factory for creating objects in the database.

> Is there any need to handle data that has been altered incorrectly on the
> file system?
 > Such as give an object a property it shouldn't have or have
> access to?

Yes. This should be handled as part of synchronization.

You raise a good poinnt about adding objects on the file system.
Clearly, we'll have to give more thought to the user interface and
semantics of this.

You also raised a good point about making synchromization fgrom the file system
to the application easier and, perhaps, automatic during development.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (888) 344-4332            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org