[Zope-dev] "ZTK" futures: one big package?

Martijn Faassen faassen at startifact.com
Wed May 13 09:29:55 EDT 2009

Chris McDonough wrote:
[snip extending configuration patterns instead of replacing wholesale]
>   Often this code makes the subframework tremendously complex, and the 
> subframework grows inappropriate dependencies along the way.  *Sometimes* the 
> situation gets so confusing for a new user, they just quit and go use something 
> else.

I am curious about your use of the word "often". I think what you 
describe has happened for the example you give, namely the 
publication/traversal stack. I think it also has happened to a certain 
extent for the user authentication story. I'd like to identify other 
such nasty grown subframeworks and you seem to know about a few more.

I think in part this happened because there wasn't enough documentation 
about wholesale replacement patterns (and not enough documentation 
period), and in part it happened because the APIs or component 
relationships were just not designed as well as we'd like (as you remark 
below), and in part it happened because people often don't *want* to do 
a wholesale replacement but exactly the kind of "only reconfigure this 
bit" you describe

> This is a pattern that happens over and over again in Zope development.  In my 
> personal opinion, the original error was trying to make the subframework 
> configurable at all.  Instead, the subframework should be replaceable easily, 
> but it should itself be relatively rigid.  At very least, for subframeworks that 
> really do require extra configuration (should be very few), this configuration 
> should be done via highly specialized ZCML directives (or grokkers), as opposed 
> to some very general adapter registration that can't be easily discerned from 
> other adapter registrations by a newbie.

I agree that specialized ZCML directives and grokkers are a good thing. 
I think we had too much of a tendency to get away from those and instead 
generalize everything. I think we should define clear Python APIs for 
particular configuration concepts, not just ZCML directives or grokkers. 
In turn the ZCML/grokker implementations can use this Python API and be 
really minimal in themselves.

That said, I do think there's implementation value in a general 
registry. It's just not a great API in most cases.

> If the subframeworks were more rigid (but replaceable), the *intent* of the 
> subframework author could be more easily discerned, and fewer people would fall 
> into the trap of adding more configuration code to a subframework instead of 
> just replacing it entirely.  And fewer people would just walk away in frustration.

While I understand the sentiment and I agree with the general advice, I 
again must note that sometimes it's useful to be able to configure a 
subframework. "everything as normal with just a few tweaks" is after all 
a good way forward in many cases.

The idea you are promoting is "clearly focused frameworks". They do one 
thing well, and are configurable in a few ways and clearly replaceable too.

> What does this have to do with packaging?  Well, currently, there's a dizzying 
> number of packages that make up "the ZTK" (nee "Zope 3").  Each of these 
> packages is a pure peer of all others in a PyPI listing with no real way to get 
> a sense of their relative importance other than performing a linear audit.

One way to get some hints is to look at a dependency diagram and see 
which ones are lower down in the stack.

> Even 
> if a user *does* do a linear search of all of them, it's still awful hard to 
> discern for some new user which ones are "important", and which ones just happen 
> to exist by some inequity of history without trying to install it.  The user 
> needs to gain some holistic knowledge of the system in order to discern the 
> important bits from these historical inequities.
> Most new users understandably just walk away from *all* Zope packages before 
> they gain this knowledge; it's just too hard for them to tell the difference 
> between the truly important and reusable bits and the stuff that just happens to 
> be packaged up and released but which is useless outside of some highly specific 
> context.  In effect, we just don't communicate *intent* very effectively in our 
> current packaging structure.

Sure, agreed.

> In my opinion, this is why a lot of Python developers who are otherwise very 
> smart have given up on trying to use Zope packages.  The time required to figure 
> out which ones are useful and which ones aren't is just too high.  It's way 
> "easier" for them to write them all off wholesale and just write what they need 
> from scratch or use somebody else's software.  Good developers tend to like 
> small, useful libraries and frameworks.

Agreed again.

> We can ameliorate the situation in a few ways:
> - We can reduce the number of distributions.
> - We can make each current Zope package distribution independently useful.
> My suggestion is that we do the former first in order to communicate *current 
> intent* (the "state of reality right now").  We can do the latter after this in 
> pieces, hopefully aided by some new developers we've picked up by making it 
> easier to find useful stuff.

In order for your plan to work I think you'd need two things ideally:

* remove a lot of the released packages from pypi. This is a no-go. 
People depend on these in a massive way.

* at least move around the existing packages in svn.zope.org to some 
kind of "random crap" area so people aren't confronted with them.

> Once we deflate our current set of packages down to a reasonable number, the 
> packages listed in PyPI will immediately start to reflect "the state of reality 
> right now". As a result, we'll hopefully be able to get some new blood in the
> form of new developers that use the smaller bits outside Zope to help us tease 
> the truly independent pieces out of the larger pile.  If we do this, at no time 
> after the deflation will PyPI listings ever as badly advertise "the state of 
> reality" as it is advertised right now, and the community will hopefully again 
> start to grow.

How do you propose we reduce the PyPI listing without breaking a lot of 
code? The other practical issue is community-oriented: we have been able 
to break so many dependencies and have an awareness of these issues 
precisely because packages *were* split up.

Meanwhile we've invested quite a bit of effort in making existing Zope 
packages at least *depend* more or less logically on other packages. 
That doesn't mean they're always, or even typically, independently 
useful in the sense you want: they're useful only if you buy into a lot 
of the Zope story. I hope we can reduce the buy-in more once we explode 
the publisher, but it'll always be that way to a certain extent.

A package that depends on 10 other packages might not be considered 
"independently useful" by many, as it assumes quite a bit of the 
framework to be present. But that doesn't mean it's useless as an 
independent package.



More information about the Zope-Dev mailing list