[Zope3-dev] Re: zope.error is a 3.5 egg, but is needed by 3.4.x releases

[Jim Fulton]

On Oct 6, 2007, at 4:56 AM, Martijn Faassen wrote:

> I think Zope 3.4 is currently not usable unless you already know  
> exactly what you're doing with egg dependencies. What you're  
> supposed to be doing isn't exactly documented anywhere.

I think you are probably right.  I think part of the problem is that  
we no longer have a clear vision for what Zope 3 is.  I don't think  
this is entirely bad, as the vision has been evolving to match  
reality.  I think most of us are settling on the idea of Zope 3 as  
"library", but we're still figuring out how to articulate and  
implement this vision.

> Zope 3 the code isn't animate, but the open source community better  
> be.

Right. I consider you to be a part of this community. Several of the  
major players in the community have been highly engaged in the  
discussions over the past several days.  I think it's safe to say  
that we're trying to make things better.

>
>>> We actually care about a Grok version as it's the main way to get  
>>> people to actually use Zope 3 stuff.
>>>
>>> We noticed this while we were going through egg dependencies by  
>>> the way, not "using a Zope 3 version".
>> I don't know what you are trying to say.
>
> Fred expressed he (and "many of us") are happy he doesn't have to  
> think about a Zope 3 version anymore. The result is, as far as I  
> can see, that *everybody* has to think about versions. If there'd  
> been some release of 3.4, we'd be able to use the versions that  
> release is using, but instead, everybody is making their own  
> collections of eggs and hoping for the best.
>
> I understand that eggs can evolve at different rates, but having to  
> determine the base from which to start ourselves wasn't exactly a  
> pleasant exercise. Since everybody will have to come up with their  
> own list I imagine others aren't exactly thrilled either.

That's why I like the idea of managed indexes, similar to the package  
repositories managed for linux distributions.  I'm hopeful that this  
approach will bear fruit soon.

...

> But if nobody in the Zope 3 community steps up and starts to  
> clearly point out what people *should* be doing, the situation will  
> continue to be unworkable. New users will not be able to adopt Zope  
> 3 at all anymore.

Agreed.  Of course, before we can point out what people "should" be  
doing, we have to figure out what that is.  To do that, people have  
to remain engaged.

> I do care about getting new users to adopt Zope 3 technologies. I  
> don't see the Zope 3 community think much about new user adoption.

Lots of people care.  Of course, most of us also have day jobs.  Many  
of us are doing the best we can.


> The only way the eggified Zope can be used reliably currently is  
> with a lot of upfront knowledge most people don't have, and a lot  
> of work to sort through the eggs. There are scripts floating around  
> to extract a list of versions from a buildout run, which helps  
> some, but initially the answer I got on #zope3-dev when asking  
> about such things was "write your own script, it's easy". I thought  
> open source communities were about sharing solutions for problems,  
> not just sharing the problems themselves.

Again, we have to have the solutions before we can share them.   
Sometimes, we arrive at these solutions through experimentation.

As always, too, we have to understand the different problem sets we  
have and the perspectives that gives us.  Many of us are building  
applications with Zope 3.  Others, like you, are building platforms.   
Solutions that work for single applications, like nailing versions in  
meta egg or buildout configuration don't work so well for platforms,  
as I think you've discovered.


> I think in order to get new users to adopt the system, besides  
> clearly documenting quite involved instructions on what they should  
> be doing, there should be a way to use Zope 3 reliably *without*  
> having to have all this knowledge and doing all this work.

Agreed.

> Anyway, I know you personally are concerned with this issue, and I  
> realize that buildout has been attempting to grow features that  
> help here. I appreciate the help.

Thanks.  Note that buildout has very much an application focus.

> The Grok project is trying to solve these issues. We published a  
> document that tells users what to do. We now publish lists of eggs  
> on a HTTP server (one per grok release). We have adjusted  
> grokproject so that a versions list will be used automatically when  
> you create a new grok project. We attempt to reach a situation  
> where users can install and use Grok and build their own  
> applications without having to spend too much thought on  
> dependencies. We aim for a situation where the community manages  
> the list of dependencies, instead of everyone for themselves.

Sounds great.  I suspect that the same or similar effort could server  
the broader Zope 3 community.

> Why did this have to happen within the Grok project? I have the  
> impression that too much, people in the Zope 3 community think that  
> as soon as they themselves and the other expert developers know  
> what to do (assimilated on the mailing list and online), this is  
> the end of the problem. We don't have a single Zope 3 release  
> anymore, but we do want people to use our code and report bugs on it.

We were supposed to. I wonder what happened to that.

> As far as I can see this *requires* a list of versions somewhere  
> that is shared between people and that can be communicated about  
> easily. It requires a *release procedure* around this list of  
> versions. As I tried to point out before long ago, the Zope 3  
> project is not somehow so special it doesn't need releases.

I think the Zope 3 project is somewhat special, which it may need  
some kind of releases. :) After all, we have releases of individual  
eggs.  I don't think it makes much sense to release the traditional  
Zope 3 application except perhaps as a testing platform.  I do think  
we need something like what linux distros provide.  We need to  
establish a better understanding of this.


>>> as we have fixed the problem with Grok. If non-Grok users are  
>>> interested in our fixes, please let us know though. We've just  
>>> made this massive investment. I'd suggest people to switch to  
>>> Grok anyway, as we actually think about this stuff and care about  
>>> having our house in order.
>> Are you suggesting that other people aren't thinking about this  
>> and don't care?
>
> Fred doesn't appear to care to think about this much, certainly,  
> and is liking it that way.

So based on a snarky comment by one person, you tar the entire  
community. I'll note, BYW, that if you had provided more details, as  
Philipp eventually did, I think Fred's response would have been more  
sympathetic. (Just guessing)

>> I hope that the proposal for setting up a known-good-set index  
>> will be helpful.  I'm sure Stephan would like to know the versions  
>> you chose. I imagine he'll be able to get those by looking at Grok.
>
> After 3 days of hard work by 2 people, we're still not done with  
> the Grok eggs. We get weird effects like final releases of packages  
> *suddenly* needing an entirely new package, between the beta and  
> the final. I'm quite bothered by this.

Me too.

> We still have quite a few dev-r515135 style eggs in the mix as well.

That's bad.

> A single known-good index, by the way, doesn't solve all problems.  
> It will evolve forward as people release newer known-good versions  
> of eggs. This means that nobody can really rely on such an index,  
> as suddenly they might be starting to use 3.6 eggs where they were  
> using 3.5 eggs. Even bugfix releases are currently hardly safe:  
> using 3.4.1 for some Zope 3 package might mean that suddenly it  
> requires an entirely new package altogether, introducing new  
> deprecation warnings and so on (as in this thread).

This is all a matter of the rules for maintaining the index and the  
care put into it.  If it is well run, a big if, then it should be as  
reliable as, say, a linux package repository. This doesn't guarantee  
that there won't be problems.

> With our Grok versions list, we publish a list *per* version of  
> Grok. I hope this is the intent for the known-good index as well.  
> If someone says they use Grok 0.11,

Is Grok 0.11 a feature release?  Would you expect the version list  
for Grok .11 to change as bug fixes are made?


> we want to know *exactly* which versions they are using without  
> requiring them to send us a long list too. One version number  
> should be enough. Our installation tools support this, and our  
> documentation documents this. I think this is the only maintainable  
> way to actually have a community developing and using a common code  
> base.

My suggestion is that there should be stable indexes for each  
"feature release".  Changes to these indexes should be made to fix  
bugs, not add new features.  Of course, there could be other less  
stable indexes for people who want them.  If you want to nail all of  
the versions for a specific release of grok, then it seems to me that  
versions in a meta egg or a buildout configuration should work well.

Jim

--
Jim Fulton
Zope Corporation