[Zope3-dev] Re: Known working sets II [was: Eggification redux]

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martijn Faassen wrote:
> Dieter Maurer wrote:
>> Martijn Faassen wrote at 2007-9-25 19:57 +0200:
>>> ...
>>> If you choose for flexibility first, people will need to think about
>>> versions all the time.
>> I follow Tres argumentation: somehow the Linux distributors
>> have this problem mostly solved:
> 
> While I don't dispute we should look at package management systems, they 
> don't have *our* problems exactly solved. I have 20 different buildouts 
> installed, which may or may use the same pool of eggs, and may use 
> different versions of the same package. I am the one who wants to have 
> the final say in what versions of packages. I want to use. A linux 
> distributor needs to have one working set of packages, instead.
> 
>>   Standard distributions come with a set of known working components
>>   and quite weak dependancy declarations.
>>
>>   When I install additional components, I will be told about
>>   potential conflicts (based on the weak dependancies) and
>>   asked what to do (install nevertheless, upgrade more things to
>>   get dependancies right or abort).
>>
>>   Usually, I do not have to worry about versions -- only
>>   occationally (when I see conflict lists) or even more rarely,
>>   when something breaks even though there has not been a conflict.
> 
> A linux distribution, unless you're on debian unstable, has quite a 
> strict control over what packages enter a common pool. They do not 
> release a newer version of a library unless they know the software that 
> depends on it either works or can be upgraded to it.

Anybody running against the Cheeseshop today is *more* on the bleeding
edge than a sysadmin whose production boxes are running 'sid':  Debian
has cultural constraits, even for that distro, which are vastly more
restricted than the Wild West which is PyPI.

The only solution I can see is to create filtered subsets / mirrors of PyPI.

> We have a situation where we have developers, not maintainers, uploading 
> new versions of packages. There will be no integrated testing done for 
> all software built on all packages in the cheeseshop. Again, I can see 
> similarities, but I don't believe linux distributions have *exactly* our 
> problems solved. Our buildouts are used as development environments, not 
>   only deployment environments.

Even folks' doing pure development need more predictability than
"whatever happens to be au courant on the Cheeseshop", made worse by the
fact that *subsequent* installations may pick up projects /
distributions different than / incompatible with those which were in
effect when earlier packages were installed.

>> We currently made bad experiences with weak dependancies.
>>
>> I see several reasons for this:
>>
>>   *  not yet ready distributions (insufficiently tested,
>>      alpha quality) have been uploaded to PyPI
>>
>>      We are now aware that we must not do things like this
> 
> Better diligence here would help. It would help me most as a framework 
> developer developing the framework - I can upgrade to newer versions of 
> my dependencies without so much stuff breaking.

Without a *huge* cultural change, I predict very low success in
persuading package authors to exercise more care in what they release to
the Cheeseshop.  I further predict that efforts to manage this problem
by adding "pinned" dependencies to individual packages will make the
problem *worse*, rather than better.

>>   *  installation tools have prefered newer versions over
>>      older ones, even when the newer versions were development/alpha
>>      while the older ones have been stable
>>
>>      The tools meanwhile have changed and stick preferably to
>>      stable versions
> 
> Sticking to stable versions helps, until a new stable version is 
> released. Then all the old stuff suddenly starts using the *new* stable 
> version, and probably break.

Exactly.  Without some way to impose a "gatekeeper" role on the package
pool from which a given deployment draws, we can't have any
deterministic outcomes when installing packages.

>>   *  The installation tools work incrementally with dependancies
>>      rather than based on a global dependancy graph.
>>
>>      This is not yet changed.
> 
>> Maybe, our bad experience are drastically reduced when the above
>> reasons are taken care of -- even with weak dependancies?
> 
> I used to believe that, but after seeing Grok 0.10 break once or twice 
> *every* week for the last month, I don't believe it anymore. We're 
> talking about the same release, breaking over and over again as 
> something goes wrong with some egg somewhere. I want these dependencies 
> pinned down hard. That said, I believe there are ways to solve these 
> problems without hardcoding them in install_requires. I hope we can have 
> the benefits of weak dependencies while having the safetly of hardcoded 
> ones. See here:

I think that replacing 'index_url' with a "gated community" of packages
is the only path to sanity here:  the contract of the Cheeseshop (share
new releases of all packages with everyone ASAP") is incompatible with
our goals ("ensure that users can install a given package and its
dependencies, and have them work").


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQFG+vfY+gerLs4ltQ4RAgS0AJ9B+q0BibgbK7EZ5LyutI0l0UyTDQCVFKMg
7cBORRpQtq4Nc1YbT9yoYg==
=ROuj
-----END PGP SIGNATURE-----