[Zope-dev] why external version indexes don't fulfill all use cases for development

[Martijn Faassen]

Hey,

On Nov 11, 2007 10:34 PM, Jim Fulton <jim at zope.com> wrote:
>
> On Nov 11, 2007, at 2:06 AM, Martijn Faassen wrote:
[snip]
>
> > This breaks a fundamental assumption for releases. When I release
> > something, I expect it to work tomorrow, next month, and next year.
>
> If you want this, then you can't rely on the KGS.  When releasing our
> applications, we don't rely on a KGS. We fix all of the versions
> we're using.  IMO, the KGS shouldn't try to solve this problem.  A
> KGS should be helpful for developers and development frameworks.  A
> KGS will be more useful if the quality remains high.   A KGS is
> similar to a traditional monolithic release.  After all, bug fix Zope
> releases have been known to break applications too.

I got completely confused by your answers you gave previously: you
were talking about feature releases, but of what?

Basically here you say that KGS replaces a monolithic release of Zope
3. I see KGS as useful for the developers of Zope 3 classic. I see KGS
as a useful source of tested lists of versions where they are related
to Zope 3.

> > With code, we know that history, and branches, and so on, are
> > important. We use Subversion. With KGS we only have an ongoing trunk.
>
> I'm not sure why you keep saying "trunk".  I'm not sure if you are
> being imprecise, or if I'm missing something.
> There's no reason a KGS couldn't be managed with a revision control
> system. That might be a very good idea.

I say this as this is the impression I get from it. Saying that a KGS
could be managed with a revision control system is nice, but can it
now? How complicated would that make it? Does it make sense to
maintain this information externally to the packages?

[snip]
> > There are some fundamental problems with external lists or indexes:
> >
> > * we need to know about the dependency of dependencies, even if we
> > never use them directly. Information hiding is broken.
>
> I'm not sure how this is a problem with version lists (external or
> otherwise) or indexes.

Dependencies of dependencies itself isn't a problem, as this
information is still in packages themselves. The versions of
dependencies isn't, and this is a problem, because I don't *want* to
know about dependencies of dependencies, or their versions. I don't
want to have to care. The packages themselves should know this.

The story for beginners wouldn't be good enough, as they'd need to
know too. With ZCML we're finally resolving this by putting this
dependency structure in ZCML. Not ideal, but at least when you include
package X which needs Y which needs Z, you don't need to manually
include the ZCML of Y and Z anymore. Now with package dependencies and
versioning, I need to make decisions on the versions of Y and Z, while
I just care about using X. I don't want to know this stuff.

A beginner can of course, if he's lucky, interact with an index like
KGS that makes decisions for them. That works until they need a
different version or different package than what is maintained in the
index. In that case, they don't want anything external to make the
decisions.

The basic thing I'd like is to just to ask the package: give me the
versions *you* think you can work with. If I'm tracking this package
in subversion or upgrade to a new release of the package, this list of
best versions might change, too. I don't want to have to know, just
like I don't want to have to know about the implementation details of
a package.

Dependencies and the versions of such are an implementation detail.
One I might on occasion like to override, just like I sometimes need
to override the implementation details of a class by subclassing, but
that should be further along the curve, not immediate. I think it
would be reasonable for packages higher up in the dependency tree to
have the ability to override version decisions made by packages lower
down (as long as there aren't any conflicts within the structure). In
these case these package explicitly decide to take over
responsibility.

> > * a single list will never do it. We intend to have many different
> > applications that may depend on different versions of packages.
> > Grok may need a newer zope.publication than your application does.
> > A Grok extension may need an even newer version than Grok does.
> > We'll be baking endless amounts of lists this way.
>
> I think each application will need to come up with a version list for
> each of it's releases.  In development, an application can use an
> index or external version list as a starting point.  For example, I
> see a KGS being useful as a (fairly) stable baseline for
> development.  When an application is ready for release, it should fix
> it's versions.  I've tried to make this easy to do with buildout.
> When you're preparing to make a release, run buildout in verbose mode
> (-v) It will print out the versions it picked in a format that is
> easily turned into a version list.

Sure, I know about all this. I just am saying that this doesn't do
enough. During development I decide I want to rely on, X, which needs
Y which needs Z. Moreover, this is not in KGS, or I need newer or
older versions than those in KGS. I rely on these purely in my own
application. I will now need to figure out that there is indeed a
dependency structure for X (I don't care about Y and Z), and I need to
figure out which versions for these would be best, and make some kind
of random guess (as the information is nowhere to be found in the
packages. If I'm lucky it's in human-readable documentation, that's
it), perhaps puzzle them out from the buildout run, and bake them into
my versions list.

I think this procedure is fundamentally wrong as it puts
responsibilities with the developer that the developer shouldn't have.
The responsibilities about the best versions that fit with X are with
the developers of X, unless I should explicitly decide to override
their decision. The only thing the developer should have to choose is
the best version of X they want to use. If I want to use an older
version of X, I should be able to do this, too.

If I have to maintain my own lists for this, the chances that these
will eventually be the wrong lists, the out of date lists, the "I
changed the version for X in my list, but oops, I was also supposed to
update Y and Z" mistakes, and so on. As far as I can see, if this
information is inside the packages themselves, I can pick any version
of the the package and have the right list, automatically. I don't
need to worry about external indexes or historical versions.

> No one is advocating a KGS for each application.  A KGS is merely an
> attempt to provide a stable ecosystem of components that work
> together.  Just like Linux is a platform for delivering applications,
> a KGS is a platform for building applications.

Yes, and I'm not debating *against* a KGS. It obviously works better
than anything else we have right now for what it does. I'm just
debating that it isn't *sufficient*. In addition, I'm debating that
what applications and frameworks are starting to do now (external
lists of versions for themselves) is also sub-optimal.

Finally, I am also wondering whether externally is really the right
place to maintain this information in the first place. That's not to
say I want KGS to disappear today, but I do want to avoid getting
locked into a strategy forever without further consideration of the
alternatives.

Regards,

Martijn