From zope-tests at epy.co.at Tue Jul 1 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Tue Jul 1 06:59:55 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807011100.m61B05xE008035@shnoll.test.at> Summary of messages to the zope-tests list. Period Mon Jun 30 11:00:00 2008 UTC to Tue Jul 1 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Mon Jun 30 20:59:46 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009785.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Mon Jun 30 21:01:16 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009786.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Mon Jun 30 21:02:47 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009787.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Mon Jun 30 21:04:17 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009788.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Mon Jun 30 21:05:47 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009789.html From gns24 at mythic-beasts.com Tue Jul 1 23:25:10 2008 From: gns24 at mythic-beasts.com (Graham Stratton) Date: Tue Jul 1 23:25:05 2008 Subject: [Zope-dev] TutorialTest with zc.testbrowser.real Message-ID: <10F26F09-C5C7-4879-931E-D16BB6E10594@mythic-beasts.com> Hi all, I currently need to write a tutorial for how to use the web application which I've been developing. I'd really like to be able to produce something testable, in a doctest-like way. My best idea so far is to build something based on zc.testbrowser.real, incorporating the ability to display comments on the screen and to pause for some amount of time/until the user clicks the mouse. The tutorial would need to run as a standard test (using zc.testbrowser.browser?) when run from the test runner, but in human- friendly mode when run as a tutorial. I'm guessing that something like this would be very useful to many people; so much so that I'd expect something similar to have been done before. Is this the case? I have concerns about getting the tutorial to run against an existing ZODB without making modifications to it. I think there's some kind of storage which can wrap an existing storage, can anyone point me the right way? I think Stephan and others had some ideas along these lines at the Foliage Sprint. I'd be grateful for your thoughts before I get started. Thanks, Graham From zope-tests at epy.co.at Wed Jul 2 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Wed Jul 2 06:59:53 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807021100.m62B05a3021915@shnoll.test.at> Summary of messages to the zope-tests list. Period Tue Jul 1 11:00:00 2008 UTC to Wed Jul 2 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Tue Jul 1 20:53:32 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009790.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 1 20:55:03 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009791.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 1 20:56:33 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009792.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 1 20:58:03 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009793.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 1 20:59:33 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009794.html From benji at zope.com Wed Jul 2 08:14:07 2008 From: benji at zope.com (Benji York) Date: Wed Jul 2 08:13:55 2008 Subject: [Zope-dev] TutorialTest with zc.testbrowser.real In-Reply-To: <10F26F09-C5C7-4879-931E-D16BB6E10594@mythic-beasts.com> References: <10F26F09-C5C7-4879-931E-D16BB6E10594@mythic-beasts.com> Message-ID: On Tue, Jul 1, 2008 at 11:25 PM, Graham Stratton wrote: > Hi all, > > I currently need to write a tutorial for how to use the web application > which I've been developing. I'd really like to be able to produce something > testable, in a doctest-like way. > > My best idea so far is to build something based on zc.testbrowser.real, Note that zc.testbrowser.real is experimental -- i.e. its API will most certainly change and it may even be discontinued entirely. > I have concerns about getting the tutorial to run against an > existing ZODB without making modifications to it. I think there's some kind > of storage which can wrap an existing storage, can anyone point me the right > way? You may be referring to demostorage (http://pypi.python.org/pypi/zc.demostorage2). -- Benji York Senior Software Engineer Zope Corporation From zope-tests at epy.co.at Thu Jul 3 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Thu Jul 3 06:59:53 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807031100.m63B05cs013883@shnoll.test.at> Summary of messages to the zope-tests list. Period Wed Jul 2 11:00:00 2008 UTC to Thu Jul 3 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Wed Jul 2 20:53:47 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009795.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 2 20:55:17 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009796.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 2 20:56:47 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009797.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 2 20:58:17 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009798.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 2 20:59:47 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009799.html From sw at gocept.com Thu Jul 3 08:15:59 2008 From: sw at gocept.com (Sebastian Wehrmann) Date: Thu Jul 3 08:15:44 2008 Subject: [Zope-dev] zc.testbrowser.real support for mozlab 0.1.9 Message-ID: <486CC2FF.6050906@gocept.com> Hi, I've checked in a branch with changes to the testbrowser.real code to make it work with mozlab 0.1.9 (and firefox 3). Could someone please review the code and inform me, if I can merge it with the trunk? The branch can be found at svn.zope.org/repos/main/zc.testbrowser/branches/sweh-mozlab0.1.9 Regards, Sebastian -- Sebastian Wehrmann ? sw@gocept.com gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany http://gocept.com ? tel +49 345 1229889 12 ? fax +49 345 1229889 1 Zope and Plone consulting and development -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4066 bytes Desc: S/MIME Cryptographic Signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080703/fb1342bd/smime.bin From benji at zope.com Thu Jul 3 08:44:03 2008 From: benji at zope.com (Benji York) Date: Thu Jul 3 08:43:50 2008 Subject: [Zope-dev] zc.testbrowser.real support for mozlab 0.1.9 In-Reply-To: <486CC2FF.6050906@gocept.com> References: <486CC2FF.6050906@gocept.com> Message-ID: On Thu, Jul 3, 2008 at 8:15 AM, Sebastian Wehrmann wrote: > Hi, > > I've checked in a branch with changes to the testbrowser.real code to > make it work with mozlab 0.1.9 (and firefox 3). Cool. > Could someone please review the code and inform me, if I can merge it > with the trunk? I'll review it in the next couple of days and get back with you. -- Benji York Senior Software Engineer Zope Corporation From benji at zope.com Thu Jul 3 17:22:11 2008 From: benji at zope.com (Benji York) Date: Thu Jul 3 17:22:00 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down Message-ID: I'm working on making the zope.testing test runner run tests in parallelized subprocesses. The option will likely be spelled -j N, where N is the maximum number of processes. I have it basically working, but have noticed a couple odd corners of the test runner that I'd like to clean up. They may be controversial, so I'll ask about them here first. I'd like to 1) remove the layer tear-down mechanism entirely, and 2) make (almost) all layers run in a subprocess. I want to do #1 because it would simplify the test runner code and no one seems to be using the functionality anyway. It also appears from reading the code that any tests run in a subprocess (and most are) will never exercise the tear-down mechanism anyway. #2 will add some process start-up overhead, but it'll only be one more process than is already started (and any reasonably-sized test corpus already starts several processes for each test run). The one exception is for running with -D or with a pdb.set_trace() embedded in the code under test. For that case we need a switch to say "don't start any subprocesses at all", I suspect that will be spelled -j0. For motivation, some speed comparisons: running a particular test suite with 3876 tests (mostly doctests, and mostly functional) without the patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 minutes and 22 seconds (give or take) on a dual-core box with 3 simultaneous subprocesses. -- Benji York Senior Software Engineer Zope Corporation From ct at gocept.com Thu Jul 3 17:37:13 2008 From: ct at gocept.com (Christian Theune) Date: Thu Jul 3 17:37:04 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: <1215121033.7325.2.camel@mindy> On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. > > I have it basically working, but have noticed a couple odd corners of > the test runner that I'd like to clean up. They may be controversial, > so I'll ask about them here first. > > I'd like to 1) remove the layer tear-down mechanism entirely, and 2) > make (almost) all layers run in a subprocess. > > I want to do #1 because it would simplify the test runner code and no > one seems to be using the functionality anyway. It also appears from > reading the code that any tests run in a subprocess (and most are) will > never exercise the tear-down mechanism anyway. +1 in general but -1 on removing the tear down functionality. We use it to destroy external databases that where generated for setup. > #2 will add some process start-up overhead, but it'll only be one more > process than is already started (and any reasonably-sized test corpus > already starts several processes for each test run). The one exception > is for running with -D or with a pdb.set_trace() embedded in the code > under test. For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. +1 as well. I'm actually wondering whether we might be able to control the pdb through a sub-process. > For motivation, some speed comparisons: running a particular test suite > with 3876 tests (mostly doctests, and mostly functional) without the > patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 > minutes and 22 seconds (give or take) on a dual-core box with 3 > simultaneous subprocesses. Yay! -- Christian Theune ? ct@gocept.com gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany http://gocept.com ? tel +49 345 1229889 7 ? fax +49 345 1229889 1 Zope and Plone consulting and development -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080703/bf1b3e89/attachment-0001.bin From dev at projekt01.ch Thu Jul 3 17:42:13 2008 From: dev at projekt01.ch (Roger Ineichen) Date: Thu Jul 3 17:41:30 2008 Subject: AW: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: Hi Benji > Betreff: [Zope-dev] Test runner: layers, subprocesses, and tear down [... ] > #2 will add some process start-up overhead, but it'll only be > one more process than is already started (and any > reasonably-sized test corpus already starts several processes > for each test run). The one exception is for running with -D > or with a pdb.set_trace() embedded in the code under test. > For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. That's a very important point. I often use pdb if I write tests. > For motivation, some speed comparisons: running a particular > test suite with 3876 tests (mostly doctests, and mostly > functional) without the patch takes 6 minutes, 42 seconds; my > branch runs the same tests in 3 minutes and 22 seconds (give > or take) on a dual-core box with 3 simultaneous subprocesses. Yeah, great! Regards Roger Ineichen > Benji York > Senior Software Engineer > Zope Corporation From benji at zope.com Thu Jul 3 17:44:49 2008 From: benji at zope.com (Benji York) Date: Thu Jul 3 17:44:38 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: <1215121033.7325.2.camel@mindy> References: <1215121033.7325.2.camel@mindy> Message-ID: On Thu, Jul 3, 2008 at 5:37 PM, Christian Theune wrote: > On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: >> I'd like to 1) remove the layer tear-down mechanism entirely, and 2) >> make (almost) all layers run in a subprocess. >> >> I want to do #1 because it would simplify the test runner code and no >> one seems to be using the functionality anyway. It also appears from >> reading the code that any tests run in a subprocess (and most are) will >> never exercise the tear-down mechanism anyway. > > +1 in general but -1 on removing the tear down functionality. We use it > to destroy external databases that where generated for setup. Ah! Good point. >> #2 will add some process start-up overhead, but it'll only be one more >> process than is already started (and any reasonably-sized test corpus >> already starts several processes for each test run). The one exception >> is for running with -D or with a pdb.set_trace() embedded in the code >> under test. For that case we need a switch to say "don't start any >> subprocesses at all", I suspect that will be spelled -j0. > > +1 as well. I'm actually wondering whether we might be able to control > the pdb through a sub-process. I don't think it'd be that hard, in general, but the current design of using stdout and stderr for IPC communication channels is a hindrance. >> For motivation, some speed comparisons: running a particular test suite >> with 3876 tests (mostly doctests, and mostly functional) without the >> patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 >> minutes and 22 seconds (give or take) on a dual-core box with 3 >> simultaneous subprocesses. > > Yay! I have an 8 core machine that I can't wait to try it on. ;) -- Benji York Senior Software Engineer Zope Corporation From marius at gedmin.as Thu Jul 3 18:50:49 2008 From: marius at gedmin.as (Marius Gedminas) Date: Thu Jul 3 18:50:37 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: <20080703225049.GA19616@musmire.b4net.lt> On Thu, Jul 03, 2008 at 05:22:11PM -0400, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. That's wonderful news! > I have it basically working, but have noticed a couple odd corners of > the test runner that I'd like to clean up. They may be controversial, > so I'll ask about them here first. > > I'd like to 1) remove the layer tear-down mechanism entirely, and 2) > make (almost) all layers run in a subprocess. -1 in general. +1 if you do that only for -j N where N > 1. Running all the tests in a single process has the following benefits: * test coverage analysis produces results that are correct (well, correct often enough -- but it has no chance at all when the test runner forks a subprocess) * import pdb; pdb.set_trace() works > I want to do #1 because it would simplify the test runner code and no > one seems to be using the functionality anyway. That's news to me. A while ago I went through Zope 3 trunk (it was pre-eggsplosion IIRC) and made sure all test layers defined in it supported teardown. Granted, FunctionalTestLayer() has allow_teardown=False as the default, for two reasons: * backwards compatibility: in olden days functional test layers didn't support teardown * paranoia: it is in general impossible to determine whether calling CleanUp().cleanUp() will correctly clear all the global state (someone could easily write a custom ZCML directive that changed a global variable and forget to register a CleanUp hook), so disallowing teardowns were the conservative safe choice. It is entirely my fault that I haven't evangelized the allow_teardown=True option for creating new test layers. > It also appears from > reading the code that any tests run in a subprocess (and most are) will > never exercise the tear-down mechanism anyway. I guess that's fine for process state, but not so fine for external state (temporary files etc.). Hey, this might explain why SchoolTool's tests tend to fill up my buildbot's /tmp without cleaning up after themselves! I'll have to investigate some day. > #2 will add some process start-up overhead, but it'll only be one more > process than is already started (and any reasonably-sized test corpus > already starts several processes for each test run). The one exception > is for running with -D or with a pdb.set_trace() embedded in the code > under test. For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. If that case needs to be supported anyway, what's the advantage of spawning exactly one subprocess when you run it with -j 1? I would also question whether pdb-unfriendly non-performance-enhancing option should be the default. > For motivation, some speed comparisons: running a particular test suite > with 3876 tests (mostly doctests, and mostly functional) without the > patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 > minutes and 22 seconds (give or take) on a dual-core box with 3 > simultaneous subprocesses. I know; for large test suites (by "large" I mean 40 minutes) I've been using an ugly hack (--odd/--even test filtering) that lets me use both CPUs if I manually run two instances of the test runner in two xterms in parallel. Regards, Marius Gedminas -- "Wipe Info uses hexadecimal values to wipe files. This provides more security than wiping with decimal values." -- Norton SystemWorks 2002 Manual -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: Digital signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/edfefc23/attachment.bin From marius at gedmin.as Thu Jul 3 19:05:30 2008 From: marius at gedmin.as (Marius Gedminas) Date: Thu Jul 3 19:05:18 2008 Subject: [Zope-dev] TALES iterator odd/even reversal Message-ID: <20080703230530.GB19616@musmire.b4net.lt> Recently I migrated a large-ish app built on Zope 3.2 to Zope 3.4. ("About time" I hear someone mumbling in the audience.) One strange difference was that TALES iterators swapped the meaning of odd and even, i.e.

odd even

produces different results on Zope 3.4 than it did on 3.2. Does anyone know why this is? Is this a bug? Should it be fixed? Marius Gedminas -- Perl is hard for most people to write. They write PERL or Pearl. -- Abigail -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: Digital signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/b0753dd1/attachment.bin From marius at gedmin.as Thu Jul 3 19:10:33 2008 From: marius at gedmin.as (Marius Gedminas) Date: Thu Jul 3 19:10:21 2008 Subject: [Zope-dev] Itemtraverser and Unauthorized vs Views In-Reply-To: <20080624113928.GG7761@mindy> References: <20080624113928.GG7761@mindy> Message-ID: <20080703231033.GC19616@musmire.b4net.lt> On Tue, Jun 24, 2008 at 01:39:28PM +0200, Christian Theune wrote: > I have a problem with the standard item traverser provided by > zope.app.container: > > The item traverser looks up a object using the given name and a __getitem__ > call on the context. If this raises a KeyError it tries to look up a view > given the same name. > > If the user does not have the permission to access __getitem__ it will let the > Unauthorized exception pass through. > > I my situation I have two views for which the user doesn't really need the > permission to access __getitem__ on the container but they can't access the > views because the __getitem__ call will be tried anyway. > > I can explicitly make the URL use '@@viewname' and bypass the item traverser, > but I don't like the @@s in the URL. I wonder whether adding Unauthorized to > the KeyError would be reasonable. I think not. At least it should not convert Unauthorized into NotFound. If I can access a location (say, http://localhost/container/item) when I'm logged in, then if I try that as an anonymous user, I should get an authentication dialog rather than a 404 Not Found page. Marius Gedminas -- If nothing else helps, read the documentation. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: Digital signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/5590bd33/attachment.bin From gns24 at mythic-beasts.com Thu Jul 3 19:19:47 2008 From: gns24 at mythic-beasts.com (Graham Stratton) Date: Thu Jul 3 19:20:06 2008 Subject: [Zope-dev] zc.testbrowser.real support for mozlab 0.1.9 In-Reply-To: <486CC2FF.6050906@gocept.com> References: <486CC2FF.6050906@gocept.com> Message-ID: <2A825211-BB78-4E77-932D-BFED09B877F7@mythic-beasts.com> > I've checked in a branch with changes to the testbrowser.real code to > make it work with mozlab 0.1.9 (and firefox 3). Hi Sebastian, This is great. I spent all day yesterday trying to make this happen and didn't get anywhere. Sometimes I wake up with a better idea as to how to solve things, but I don't often wake up to find all the work's been done! I tried running the tests with Python 2.4. About half the time they pass fine, and the other half fail something like this: File "", line 1, in ? browser.open('navigate.html') File "/Users/graham/development/patched_zc.testbrowser/src/zc/ testbrowser/real.py", line 201, in open self.wait() File "/Users/graham/development/patched_zc.testbrowser/src/zc/ testbrowser/real.py", line 193, in wait assert self.execute('tb_page_loaded;') == 'false' AssertionError though for varying pages with browser.open(). I don't understand what could be causing it; why would that value be changed in between those two lines? I guess the time.sleep() on line 191 implies that there's something we might want to wait for. I increased it by 10x and only got 1 failure out of 10, so I guess that helps. Thanks, Graham From optilude at gmx.net Thu Jul 3 19:56:28 2008 From: optilude at gmx.net (Martin Aspeli) Date: Thu Jul 3 19:59:55 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. I have some recent experience parallelising (and distributing across machines) test runs. This was in Java, with TestNG and Selenium, but we learned some interesting things. We basically cut a 45 minute test run to 10 minutes by distributing the tests across three machines, each running a full stack (Oracle, JBoss, Firefox) and Selenium Grid. I realise you're not trying to do anything quite as complex as that, but a parallel test runner ought to be extensible to support distribution across nodes in a grid. The main challenge there is to distribute deployment of the code to run, and to sync test setup so that all environments are identical. I suspect you'll find this out of scope to begin with, but I'd keep it in the back of your mind. You will likely need some way of declaring tests that have to run in series. Sometimes that's just for sanity's sake, other times it's a requirement due to shared resources. A nice way to do this is to make it possible to annotate tests to group them, and then to be able to declaratively configure some groups as serial. Any functional test that uses a shared external resource will require this. TestNG supports (as far as I recall): - Run all tests (methods) randomly and parallelise - Run groups of tests (classes or declaratively specified named groups) in parallel, but run tests within the groups sequentially - Run all tests in series (i.e. single-threaded) We should probably use test layers as the main grouping mechanism here. If you could declare a layer as "can be run in parallel with other layers" or "tests in this layer run in series", that'd be pretty powerful. I'm not 100% sure how this works with layers that derive from one another, and where you'd have two layers with a shared base class, though. Parallisation can offer huge (!) speed increases, but it can also be hard to debug tests. I'd be tempted to let single threaded by the default, safe choice, and let people opt into parallisation only when they know what they are doing. Most test runs are quite quick anyway. Test result reporting can be difficult. You'll probably need to collect all failures with tracebacks and report at the end. For long running test suites, this may not be ideal, since it's helpful to get early warning, so if you can find a way to get test output to be atomically output, then that'd be nice. Debugging stuff that happens in parallel with pdb is also tricky. It must be easy to turn off parallel running and to run individual tests in a single process for each debugging. To make this work with Selenium grid, we ended up building some infrastructure to manage environments (i.e. an allocation of database, web server and so on), and locks on those environments. We'd spawn one thread for each environment and feed tests to those threads as fast as they could run them. Each test run then grabbed an environment on setup, executed, and then released the lock for another test. Oh, and please don't get rid of any tear-down. You'll definitely need it one day. Letting environments go dirty is generally troublesome, and gets only more difficult when you may have multiple threads trying to use those environments at once. I don't know how you've structured this, but I'd consider whether one layer could be shared across multiple threads/subprocesses, or if it's always a one-to-one thing. I realise this is somewhat rambling, but I hope it's useful in any case. :) Cheers, Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book From ct at gocept.com Fri Jul 4 01:37:48 2008 From: ct at gocept.com (Christian Theune) Date: Fri Jul 4 01:37:38 2008 Subject: [Zope-dev] Itemtraverser and Unauthorized vs Views In-Reply-To: <20080703231033.GC19616@musmire.b4net.lt> References: <20080624113928.GG7761@mindy> <20080703231033.GC19616@musmire.b4net.lt> Message-ID: <1215149868.7325.7.camel@mindy> On Fri, 2008-07-04 at 02:10 +0300, Marius Gedminas wrote: > On Tue, Jun 24, 2008 at 01:39:28PM +0200, Christian Theune wrote: > > [...] > > I can explicitly make the URL use '@@viewname' and bypass the item traverser, > > but I don't like the @@s in the URL. I wonder whether adding Unauthorized to > > the KeyError would be reasonable. > > I think not. At least it should not convert Unauthorized into NotFound. > > If I can access a location (say, http://localhost/container/item) when > I'm logged in, then if I try that as an anonymous user, I should get an > authentication dialog rather than a 404 Not Found page. Actually, in my case its, when logged in I can use: http://localhost/container/view When not logged in, I get an Unauthorized, although when accessing http://localhost/container/@@view I can go ahead as anonymous. IMHO the code merging the namespaces should be more careful about that. Christian -- Christian Theune ? ct@gocept.com gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany http://gocept.com ? tel +49 345 1229889 7 ? fax +49 345 1229889 1 Zope and Plone consulting and development -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/cef09b08/attachment-0001.bin From agroszer at gmail.com Fri Jul 4 03:02:30 2008 From: agroszer at gmail.com (Adam GROSZER) Date: Fri Jul 4 03:02:50 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: <1608675482.20080704090230@gmail.com> Hello Benji, +1 for keeping the default as no subprocess and keeping the teardown. The others already said the reasons. -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: It is a great mistake to suppose that God is only, or even chiefly, concerned with religion. - William Temple, Archbishop of Canterbury From cz at gocept.com Fri Jul 4 03:34:26 2008 From: cz at gocept.com (Christian Zagrodnick) Date: Fri Jul 4 03:34:26 2008 Subject: [Zope-dev] Re: zc.testbrowser.real support for mozlab 0.1.9 References: <486CC2FF.6050906@gocept.com> <2A825211-BB78-4E77-932D-BFED09B877F7@mythic-beasts.com> Message-ID: On 2008-07-04 01:19:47 +0200, Graham Stratton said: >> I've checked in a branch with changes to the testbrowser.real code to >> make it work with mozlab 0.1.9 (and firefox 3). > > Hi Sebastian, > > This is great. I spent all day yesterday trying to make this happen > and didn't get anywhere. Sometimes I wake up with a better idea as to > how to solve things, but I don't often wake up to find all the work's > been done! > > I tried running the tests with Python 2.4. About half the time they > pass fine, and the other half fail something like this: > > File "", line 1, in ? > browser.open('navigate.html') > File "/Users/graham/development/patched_zc.testbrowser/src/zc/ > testbrowser/real.py", line 201, in open > self.wait() > File "/Users/graham/development/patched_zc.testbrowser/src/zc/ > testbrowser/real.py", line 193, in wait > assert self.execute('tb_page_loaded;') == 'false' > AssertionError > > though for varying pages with browser.open(). I don't understand what > could be causing it; why would that value be changed in between those > two lines? I guess the time.sleep() on line 191 implies that there's > something we might want to wait for. I increased it by 10x and only > got 1 failure out of 10, so I guess that helps. I had this error, too and checked in a maybe-fix in r87964. Have you tried with this revision or the one before? If it's thisone I'll add another loop there. -- Christian Zagrodnick ? cz@gocept.com gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany http://gocept.com ? tel +49 345 1229889 4 ? fax +49 345 1229889 1 Zope and Plone consulting and development From markus.kemmerling at meduniwien.ac.at Fri Jul 4 03:37:15 2008 From: markus.kemmerling at meduniwien.ac.at (Markus Kemmerling) Date: Fri Jul 4 03:37:06 2008 Subject: [Zope-dev] Itemtraverser and Unauthorized vs Views In-Reply-To: <1215149868.7325.7.camel@mindy> References: <20080624113928.GG7761@mindy> <20080703231033.GC19616@musmire.b4net.lt> <1215149868.7325.7.camel@mindy> Message-ID: <3A2499A2-E115-439D-8E29-D87DD68C11EC@meduniwien.ac.at> Am 04.07.2008 um 07:37 schrieb Christian Theune: > On Fri, 2008-07-04 at 02:10 +0300, Marius Gedminas wrote: >> On Tue, Jun 24, 2008 at 01:39:28PM +0200, Christian Theune wrote: >>> [...] >>> I can explicitly make the URL use '@@viewname' and bypass the >>> item traverser, >>> but I don't like the @@s in the URL. I wonder whether adding >>> Unauthorized to >>> the KeyError would be reasonable. >> >> I think not. At least it should not convert Unauthorized into >> NotFound. >> >> If I can access a location (say, http://localhost/container/item) >> when >> I'm logged in, then if I try that as an anonymous user, I should >> get an >> authentication dialog rather than a 404 Not Found page. > > Actually, in my case its, when logged in I can use: > > http://localhost/container/view > > When not logged in, I get an Unauthorized, although when accessing > > http://localhost/container/@@view > > I can go ahead as anonymous. > > IMHO the code merging the namespaces should be more careful about > that. IMHO the ItemTraverser should not lookup the view by itself, but delegate to the 'view' traverser, somethind like: def publishTraverse(self, request, name): """See zope.publisher.interfaces.IPublishTraverse""" try: return self.context[name] except KeyError: try: return namespaceLookup('view', name, self.context, request) except TraversalError: pass raise NotFound(self.context, name, request) Regards Markus Kemmerling From zope-tests at epy.co.at Fri Jul 4 07:00:06 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Fri Jul 4 06:59:54 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807041100.m64B06un023986@shnoll.test.at> Summary of messages to the zope-tests list. Period Thu Jul 3 11:00:00 2008 UTC to Fri Jul 4 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Thu Jul 3 20:53:07 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009800.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Thu Jul 3 20:54:38 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009801.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Thu Jul 3 20:56:08 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009802.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Thu Jul 3 20:57:38 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009803.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Thu Jul 3 20:59:08 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009804.html From sw at gocept.com Fri Jul 4 09:49:32 2008 From: sw at gocept.com (Sebastian Wehrmann) Date: Fri Jul 4 09:49:27 2008 Subject: [Zope-dev] zc.testbrowser.real support for mozlab 0.1.9 In-Reply-To: <486CC2FF.6050906@gocept.com> References: <486CC2FF.6050906@gocept.com> Message-ID: <486E2A6C.4090006@gocept.com> While using the zc.testbrowser.real testbrowser we encountered a 'bug' in browser.contents. The doctype and html-tags are swallowed. I submitted a bug fix in #87999 on my branch. Regards, Sebastian -- Sebastian Wehrmann ? sw@gocept.com gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany http://gocept.com ? tel +49 345 1229889 12 ? fax +49 345 1229889 1 Zope and Plone consulting and development -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4066 bytes Desc: S/MIME Cryptographic Signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/6e62737f/smime.bin From benji at zope.com Fri Jul 4 11:50:34 2008 From: benji at zope.com (Benji York) Date: Fri Jul 4 11:50:21 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: On Thu, Jul 3, 2008 at 5:22 PM, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. The branch (svn+ssh://svn.zope.org/repos/main/zope.testing/branches/benji-parallelize-subprocesses) is feature complete. I basically did a very simple transformation that resulted in the runner spawning subprocesses in threads, several at a time, instead of spawning them serially. The patch is less than 250 lines. Any critiques of the changes would be appreciated. No changes to the tear-down mechanism were made. All existing tests pass without modification. There aren't yet any tests for the new functionality. It may be tricky to test, so I have to think about that bit. I found that to eliminate nearly all CPU idle time, I had to use -j4 on my two core laptop. For a particular test corpus on a 4 core machine -j1 (the default) takes about 7 minutes -j6 takes about 2 minutes 20 seconds. If you use zc.buildout, then you can try the branch by checking it out, adding a "develop" entry into your buildout config referencing it, and updating any version spec for zope.testing to "3.6dev". I'd really like third-party confirmation of the total test time reductions I've seen. -- Benji York Senior Software Engineer Zope Corporation From lists at zopyx.com Fri Jul 4 12:05:05 2008 From: lists at zopyx.com (Andreas Jung) Date: Fri Jul 4 12:05:00 2008 Subject: [Zope-dev] TALES iterator odd/even reversal In-Reply-To: <20080703230530.GB19616@musmire.b4net.lt> References: <20080703230530.GB19616@musmire.b4net.lt> Message-ID: <466D0A31D3FDF968A45ABB9B@[192.168.0.24]> --On 4. Juli 2008 02:05:30 +0300 Marius Gedminas wrote: > Recently I migrated a large-ish app built on Zope 3.2 to Zope 3.4. > ("About time" I hear someone mumbling in the audience.) One strange > difference was that TALES iterators swapped the meaning of odd and even, > i.e. > >

> odd > even >

> > produces different results on Zope 3.4 than it did on 3.2. First, your example does not work (must be repeat/item instead of repeat/var). > > Does anyone know why this is? Is this a bug? Should it be fixed? Where exactly is the bug? The output is even-odd-even-odd. The even/odd methods apply to the current iteration number afaik - but not to the variable within the current iteration itself. Or? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/992b9fd5/attachment.bin From plone at hannosch.info Fri Jul 4 13:12:47 2008 From: plone at hannosch.info (Hanno Schlichting) Date: Fri Jul 4 13:12:46 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: <486E5A0F.60003@hannosch.info> Hi. Benji York wrote: > If you use zc.buildout, then you can try the branch by checking it out, > adding a "develop" entry into your buildout config referencing it, and > updating any version spec for zope.testing to "3.6dev". I'd really like > third-party confirmation of the total test time reductions I've seen. As this sounds very cool, I couldn't help but try it. I do get an error, though: Ran 80 tests with 2 failures and 0 errors in 37.463 seconds. Exception in thread Thread-1: Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/threading.py", line 442, in __bootstrap self.run() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/threading.py", line 422, in run self.__target(*self.__args, **self.__kwargs) File "/opt/tmp/zope.testing/src/zope/testing/testrunner/runner.py", line 419, in spawn_layer_in_subprocess raise SubprocessError( SubprocessError: No subprocess summary found: Error: option --resume-layer not recognized This is inside a test collection in the Plone land, so there might be something on the various other layers involved here, that causes this. But maybe you have an idea what this is about? Hanno From chris at simplistix.co.uk Fri Jul 4 14:12:47 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri Jul 4 14:12:22 2008 Subject: [Zope-dev] Deprecation warnings from 2.11.0 In-Reply-To: References: <891A9FDE8A80387848366BBA@192.168.0.24> <4860BD8A.6030104@simplistix.co.uk> <48620149.5080702@simplistix.co.uk> Message-ID: <486E681F.1040505@simplistix.co.uk> Sidnei da Silva wrote: > On 6/25/08, Chris Withers wrote: >> Cool. Are the deprecation warning common to Zope 2.11 across all platforms? > > I don't see a reason why not, specially if coming from that module. Did we as a community really release a stable version that emits deprecation warnings?! :-( Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Jul 4 14:15:52 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri Jul 4 14:15:26 2008 Subject: [Zope-dev] Zope 3 on Python 2.5, Zope 3 releases In-Reply-To: References: Message-ID: <486E68D8.2000907@simplistix.co.uk> Martijn Faassen wrote: > Hi there, > > I'm curious about the plans of Zope 3 on Python 2.5. Me too, and I have been for ages, but I've gotten nowhere :-( > * Are people using Zope 3 with Python 2.5 already? What are your > experiences? Apparently, it's all good to go, even RestrictedPython. The thing stopping me is that no-one seems willing and able (in that combination, I'm willing but don't have the right c-compiler) to build Windows binaries and get them up on PyPI. I'll ask again: what can I do to help with this? I particularly care about: zope.security zope.component zope.interface RestrictedPython ...and any dependencies. > * Are there plans for a release after Zope 3.4? I'm just assuming that > Zope 3.4.0c1 is actually the only release that Zope 3.4 is going to see. > Any plans or interest in a Zope 3.5? I don't know what these mean anymore. With smaller eggs, surely it's just a case of getting compatible versions of the smaller libraries. And surely each egg's setup.py will make sure this happens? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Jul 4 14:24:08 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri Jul 4 14:23:43 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: <486E6AC8.5010303@simplistix.co.uk> Hi Benji, I've read the whole thread to date but thought I'd reply here... Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. Cool :-) But please defauult to 1 for backwards compatability... > I'd like to 1) remove the layer tear-down mechanism entirely, and 2) > make (almost) all layers run in a subprocess. -lots to both of these I'm afraid. I've used tear-downs extensively for everything from shutting database connections to aborting transactions to dumping DemoStorages. I'm sure I'm not the only one.. As for all layers in a sub-process, I worry that this would break existing tests in some kind of horrible nasty way... > I want to do #1 because it would simplify the test runner code and no > one seems to be using the functionality anyway. It also appears from > reading the code that any tests run in a subprocess (and most are) will > never exercise the tear-down mechanism anyway. So I guess we're not currently running tests in a sub-process? My take on the pre-refactor testrunner was that a sub-process was only used when the testrunner was testing itself? > #2 will add some process start-up overhead, but it'll only be one more > process than is already started (and any reasonably-sized test corpus > already starts several processes for each test run). The one exception > is for running with -D or with a pdb.set_trace() embedded in the code > under test. For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. Right, I use this a lot. I guess -j0 should be the default for backwards compatability? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From marius at gedmin.as Fri Jul 4 16:02:37 2008 From: marius at gedmin.as (Marius Gedminas) Date: Fri Jul 4 16:02:26 2008 Subject: [Zope-dev] TALES iterator odd/even reversal In-Reply-To: <466D0A31D3FDF968A45ABB9B@[192.168.0.24]> References: <20080703230530.GB19616@musmire.b4net.lt> <466D0A31D3FDF968A45ABB9B@[192.168.0.24]> Message-ID: <20080704200236.GA27948@musmire.b4net.lt> On Fri, Jul 04, 2008 at 06:05:05PM +0200, Andreas Jung wrote: > --On 4. Juli 2008 02:05:30 +0300 Marius Gedminas wrote: > > >Recently I migrated a large-ish app built on Zope 3.2 to Zope 3.4. > >("About time" I hear someone mumbling in the audience.) One strange > >difference was that TALES iterators swapped the meaning of odd and even, > >i.e. > > > >

> > odd > > even > >

> > > >produces different results on Zope 3.4 than it did on 3.2. > > First, your example does not work (must be repeat/item instead of > repeat/var). Right, sorry about that. > >Does anyone know why this is? Is this a bug? Should it be fixed? > > Where exactly is the bug? Zope 3.2 produces odd even odd even Zope 3.4 produces even odd even odd i.e. the meanings of "odd" and "even" have switched places. As a more concrete example, when renderin a table, the first row used to get the "odd" CSS class, but now is getting the "even" one. > The output is even-odd-even-odd. The even/odd methods apply to the current > iteration number afaik - but not to the variable within the current > iteration itself. Or? Right. I'm now regretting my unfortunate example that used numbers as the items to be iterated over. Marius Gedminas -- Read what I mean, not what I write. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: Digital signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/f4fa7bb4/attachment.bin From marius at gedmin.as Fri Jul 4 16:49:43 2008 From: marius at gedmin.as (Marius Gedminas) Date: Fri Jul 4 16:49:33 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: <20080704204943.GB27948@musmire.b4net.lt> On Fri, Jul 04, 2008 at 11:50:34AM -0400, Benji York wrote: > On Thu, Jul 3, 2008 at 5:22 PM, Benji York wrote: > > I'm working on making the zope.testing test runner run tests in > > parallelized subprocesses. The option will likely be spelled -j N, > > where N is the maximum number of processes. > > The branch (svn+ssh://svn.zope.org/repos/main/zope.testing/branches/benji-parallelize-subprocesses) > is feature complete. I basically did a very simple transformation that > resulted in the runner spawning subprocesses in threads, several at a > time, instead of spawning them serially. The patch is less than 250 > lines. Any critiques of the changes would be appreciated. I'll try to take a look. > I found that to eliminate nearly all CPU idle time, I had to use -j4 on > my two core laptop. > > For a particular test corpus on a 4 core machine -j1 (the default) takes > about 7 minutes -j6 takes about 2 minutes 20 seconds. I tried this in a Zope 3.4 checkout I had handy on a Core 2 Duo machine (1.8 GHz, running 64-bit Ubuntu Hardy). One test module could not be loaded, which explains the slightly lower number of tests reported: Test-module import failures: Module: zope.app.twisted.ftp.tests.test_zopetrial Traceback (most recent call last): File "/home/mg/src/ivija-zope-3.4/Zope3.4/src/zope/app/twisted/ftp/tests/test_zopetrial.py", line 37, in ? orig_configure_logging = zope.testing.testrunner.configure_logging AttributeError: 'module' object has no attribute 'configure_logging' Here are the results: time # tests real user system reported old test runner 3m16.033s 2m44.670s 0m2.832s 6895 zope.testing trunk 2m27.816s 1m58.971s 0m2.196s 6890 new test runner -j0 2m37.322s 2m5.808s 0m2.944s 6890 new test runner -j1 2m32.249s 1m58.847s 0m2.652s 6890 new test runner -j2 2m22.287s 3m51.214s 0m13.457s 584 new test runner -j3 2m20.560s 3m46.990s 0m12.613s 584 new test runner -j4 2m30.026s 3m43.198s 0m13.241s 584 At the end of the experiment I discovered that I have CPU frequency scaling enabled. It only scales down to 1.6 GHz and quickly jumps back up to 1.87 GHz. I find the speedup by switching to a modern test runner somewhat unexpected. Can those 5 missing tests really account for 45 seconds? Zope 3 appears to be composed of a multitude of small tests. If my numbers are correct, the advantage of using both CPU cores is almost completely negated by the extra bookkeeping that the test runner has to do. Visual ogling of my CPU usage applet shows that -j0/1 use only one CPU, while -j2 and above use only one CPU for the first test layer (zope.app.apidoc.testing.APIDocLayer) and then use both CPUs for the rest. Bug? The total number of tests is misreported when you have -jN with N > 1. "Test-module import failures" is printed several times. test -j4 printed that message 37 times! test -j1 only did it once. -j2 and -j3 also did that a bit often (once per layer?) As far as I can understand, the granularity of the test distribution to CPUs is a test layer? If so, that's rather unfortunate for my application, which has only two layers (unit and functional). Especially given the quirk that the first test layer is run on one CPU while the other idles. Marius Gedminas P.S. Zope 3 is such a sweet little thing! All the tests finish in 3 minutes! Heaven. -- The planning fallacy is that people think they can plan, ha ha. -- Eliezer Yudkowsky, http://www.overcomingbias.com/2007/09/planning-fallac.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: Digital signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/b91f4c4c/attachment.bin From benji at zope.com Fri Jul 4 17:44:12 2008 From: benji at zope.com (Benji York) Date: Fri Jul 4 17:44:00 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: <20080704204943.GB27948@musmire.b4net.lt> References: <20080704204943.GB27948@musmire.b4net.lt> Message-ID: On Fri, Jul 4, 2008 at 4:49 PM, Marius Gedminas wrote: > I tried this in a Zope 3.4 checkout I had handy on a Core 2 Duo machine > (1.8 GHz, running 64-bit Ubuntu Hardy). One test module could not be > loaded, which explains the slightly lower number of tests reported: > Here are the results: > > time # tests > real user system reported > old test runner 3m16.033s 2m44.670s 0m2.832s 6895 > zope.testing trunk 2m27.816s 1m58.971s 0m2.196s 6890 > new test runner -j0 2m37.322s 2m5.808s 0m2.944s 6890 > new test runner -j1 2m32.249s 1m58.847s 0m2.652s 6890 > new test runner -j2 2m22.287s 3m51.214s 0m13.457s 584 > new test runner -j3 2m20.560s 3m46.990s 0m12.613s 584 > new test runner -j4 2m30.026s 3m43.198s 0m13.241s 584 I'm really curious why you didn't see more improvement. > At the end of the experiment I discovered that I have CPU frequency > scaling enabled. It only scales down to 1.6 GHz and quickly jumps back > up to 1.87 GHz. > > I find the speedup by switching to a modern test runner somewhat > unexpected. Can those 5 missing tests really account for 45 seconds? > > Zope 3 appears to be composed of a multitude of small tests. If my > numbers are correct, the advantage of using both CPU cores is almost > completely negated by the extra bookkeeping that the test runner has to > do. There's no appreciable bookkeeping for the parallelization, so I don't know where the CPU time is going. > Visual ogling of my CPU usage applet shows that -j0/1 use only one > CPU, while -j2 and above use only one CPU for the first test layer > (zope.app.apidoc.testing.APIDocLayer) and then use both CPUs for the > rest. Bug? Long story short: it made the changes to the code much less invasive to do that way. > The total number of tests is misreported when you have -jN with N > 1. I haven't seen that symptom, but I'll try to reproduce it by running the 3.4 tests. > "Test-module import failures" is printed several times. test -j4 > printed that message 37 times! test -j1 only did it once. -j2 and -j3 > also did that a bit often (once per layer?) Interesting. I'll investigate. > As far as I can understand, the granularity of the test distribution to > CPUs is a test layer? Right. > If so, that's rather unfortunate for my application, which has only > two layers (unit and functional). Especially given the quirk that the > first test layer is run on one CPU while the other idles. If the need is great enough, you can always introduce an arbitrary number of layers. Also, once this code is working properly, I (or someone else) might look into changing the granularity to the level of individual tests. -- Benji York Senior Software Engineer Zope Corporation From marius at gedmin.as Fri Jul 4 18:43:30 2008 From: marius at gedmin.as (Marius Gedminas) Date: Fri Jul 4 18:43:23 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: References: <20080704204943.GB27948@musmire.b4net.lt> Message-ID: <20080704224330.GA4498@musmire.b4net.lt> On Fri, Jul 04, 2008 at 05:44:12PM -0400, Benji York wrote: > On Fri, Jul 4, 2008 at 4:49 PM, Marius Gedminas wrote: > > I tried this in a Zope 3.4 checkout I had handy on a Core 2 Duo machine > > (1.8 GHz, running 64-bit Ubuntu Hardy). One test module could not be > > loaded, which explains the slightly lower number of tests reported: > > > Here are the results: > > > > time # tests > > real user system reported > > old test runner 3m16.033s 2m44.670s 0m2.832s 6895 > > zope.testing trunk 2m27.816s 1m58.971s 0m2.196s 6890 > > new test runner -j0 2m37.322s 2m5.808s 0m2.944s 6890 > > new test runner -j1 2m32.249s 1m58.847s 0m2.652s 6890 > > new test runner -j2 2m22.287s 3m51.214s 0m13.457s 584 > > new test runner -j3 2m20.560s 3m46.990s 0m12.613s 584 > > new test runner -j4 2m30.026s 3m43.198s 0m13.241s 584 > > I'm really curious why you didn't see more improvement. I wish one of the system-wide profilers (oprofile, sysprof) had support for extracting Python tracebacks out of C-level stack frames... > > Zope 3 appears to be composed of a multitude of small tests. If my > > numbers are correct, the advantage of using both CPU cores is almost > > completely negated by the extra bookkeeping that the test runner has to > > do. > > There's no appreciable bookkeeping for the parallelization, so I don't > know where the CPU time is going. Every layer is spawned in a separate subprocess, right? That means 36 new Python processes with the associated startup cost, plus the module import cost, plus some test result marshalling through plain-text Unix pipes. Two seconds of startup cost per subprocess would nicely account for the one extra minute of user time if there are over 30 subprocesses. My crude measurements (time ./test.py --list-tests > /dev/null) indicate the time needed to import everything is closer to 4 seconds, but that's importing everything -- importing just the things needed for a single layer may reduce that to two seconds on average. > > "Test-module import failures" is printed several times. test -j4 > > printed that message 37 times! test -j1 only did it once. -j2 and -j3 > > also did that a bit often (once per layer?) > > Interesting. I'll investigate. It corroborates my theory that each subprocess imports all the test modules. Marius Gedminas -- H.323 has much in common with other ITU-T standards - it features a complex binary wire protocol, a nightmarish implementation, and a bulk that can be used to fell medium-to-large predatory animals. -- Anthony Baxter -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: Digital signature Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080705/6d8847ec/attachment.bin From benji at zope.com Fri Jul 4 20:24:17 2008 From: benji at zope.com (Benji York) Date: Fri Jul 4 20:24:06 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: <20080704224330.GA4498@musmire.b4net.lt> References: <20080704204943.GB27948@musmire.b4net.lt> <20080704224330.GA4498@musmire.b4net.lt> Message-ID: On Fri, Jul 4, 2008 at 6:43 PM, Marius Gedminas wrote: > On Fri, Jul 04, 2008 at 05:44:12PM -0400, Benji York wrote: >> There's no appreciable bookkeeping for the parallelization, so I don't >> know where the CPU time is going. > > Every layer is spawned in a separate subprocess, right? That means 36 > new Python processes with the associated startup cost, plus the module > import cost, plus some test result marshalling through plain-text Unix > pipes. Two seconds of startup cost per subprocess would nicely account > for the one extra minute of user time if there are over 30 subprocesses. The number of subprocesses is the same as for the trunk, the only change is that they can be spawned in parallel. Wait! The Zope 3 test layers can all be torn down! Therefore there aren't *any* subprocesses spawned normally. Ok, that makes more sense. (time passes) OK, I did a check out of the Zope 3 trunk and was able to duplicate your results. (And wow, the trunk seems to be in bad shape -- lots of tests failing. I guess it's fallen into disrepair since being broken out into subprojects.) > My crude measurements (time ./test.py --list-tests > /dev/null) indicate > the time needed to import everything is closer to 4 seconds, but that's > importing everything -- importing just the things needed for a single > layer may reduce that to two seconds on average. A possible enhancement would be to reuse subprocesses if they are asked to run layers that can be torn down. That way for a very tear-down friendly set of tests like Zope 3's, the minimum number of processes would be started. We could also use fork to eliminate some of the start-up cost, but that's not real attractive, being un-Windows-friendly. -- Benji York Senior Software Engineer Zope Corporation From benji at zope.com Fri Jul 4 21:38:35 2008 From: benji at zope.com (Benji York) Date: Fri Jul 4 21:38:25 2008 Subject: [Zope-dev] Re: Test runner: layers, subprocesses, and tear down In-Reply-To: <20080704204943.GB27948@musmire.b4net.lt> References: <20080704204943.GB27948@musmire.b4net.lt> Message-ID: On Fri, Jul 4, 2008 at 4:49 PM, Marius Gedminas wrote: > Here are the results: > > time # tests > real user system reported > old test runner 3m16.033s 2m44.670s 0m2.832s 6895 > zope.testing trunk 2m27.816s 1m58.971s 0m2.196s 6890 > new test runner -j0 2m37.322s 2m5.808s 0m2.944s 6890 > new test runner -j1 2m32.249s 1m58.847s 0m2.652s 6890 > new test runner -j2 2m22.287s 3m51.214s 0m13.457s 584 > new test runner -j3 2m20.560s 3m46.990s 0m12.613s 584 > new test runner -j4 2m30.026s 3m43.198s 0m13.241s 584 I figured out why the total test count went down so much for -j2: the branch inherited a bug from the trunk that skips the unit test "layer" if it is run in a subprocess. I'll take a crack at fixing that. -- Benji York Senior Software Engineer Zope Corporation From ct at gocept.com Sat Jul 5 03:18:33 2008 From: ct at gocept.com (Christian Theune) Date: Sat Jul 5 03:18:23 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: References: Message-ID: <1215242313.6518.5.camel@mindy> Hi, On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. Getting back to the idea about parallelizing on a per-test base and not per-layer: The ZODB currently runs only unit tests (which became a true layer in zope.testing/trunk) but takes about XX minutes on one of my machines (4 core XEON, 3.2 GHz). I'd suggest that the general principle of splitting up the runs over multiple parallel processes should happen in a way that if you have X total tests and N parallel processes, we should have roughly X/N tests run in parallel. We could use layers as a hint to create subprocesses, but should split up layers if they are too large to fit the X/N rule (maybe with a margin of a few percent to avoid splits for single or few tests). Christian -- Christian Theune ? ct@gocept.com gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany http://gocept.com ? tel +49 345 1229889 7 ? fax +49 345 1229889 1 Zope and Plone consulting and development -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080705/0a3ccbcb/attachment.bin From ct at gocept.com Sat Jul 5 03:25:57 2008 From: ct at gocept.com (Christian Theune) Date: Sat Jul 5 03:25:47 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: <1215242313.6518.5.camel@mindy> References: <1215242313.6518.5.camel@mindy> Message-ID: <1215242757.6518.7.camel@mindy> On Sat, 2008-07-05 at 09:18 +0200, Christian Theune wrote: > Hi, > > On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: > > I'm working on making the zope.testing test runner run tests in > > parallelized subprocesses. The option will likely be spelled -j N, > > where N is the maximum number of processes. > > Getting back to the idea about parallelizing on a per-test base and not > per-layer: > > The ZODB currently runs only unit tests (which became a true layer in > zope.testing/trunk) but takes about XX minutes on one of my machines (4 > core XEON, 3.2 GHz). The actual numbers are: Ran 2816 tests with 0 failures and 0 errors in 14 minutes 34.292 seconds. real 14m36.099s user 3m44.740s sys 0m43.170s -- Christian Theune ? ct@gocept.com gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany http://gocept.com ? tel +49 345 1229889 7 ? fax +49 345 1229889 1 Zope and Plone consulting and development -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080705/c105acaf/attachment.bin From zope-tests at epy.co.at Sat Jul 5 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Sat Jul 5 06:59:53 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807051100.m65B05sD028822@shnoll.test.at> Summary of messages to the zope-tests list. Period Fri Jul 4 11:00:00 2008 UTC to Sat Jul 5 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Fri Jul 4 20:55:41 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009805.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Fri Jul 4 20:57:11 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009806.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Fri Jul 4 20:58:41 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009807.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Fri Jul 4 21:00:12 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009808.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Fri Jul 4 21:01:42 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009809.html From benji at zope.com Sat Jul 5 16:12:33 2008 From: benji at zope.com (Benji York) Date: Sat Jul 5 16:12:26 2008 Subject: [Zope-dev] Test runner: layers, subprocesses, and tear down In-Reply-To: <1215242313.6518.5.camel@mindy> References: <1215242313.6518.5.camel@mindy> Message-ID: On Sat, Jul 5, 2008 at 3:18 AM, Christian Theune wrote: > We could use layers as a hint to create subprocesses, but should split > up layers if they are too large to fit the X/N rule (maybe with a margin > of a few percent to avoid splits for single or few tests). It probably wouldn't be too hard to automatically break "large" layers into several small layers. It's also not hard for people with very large layers that care about parallel execution time to break them up themselves. I'm not opposed to automatic layer segmentation (as long as it's implemented well), but also don't think it's all that important. -- Benji York Senior Software Engineer Zope Corporation From srichter at cosmos.phy.tufts.edu Sat Jul 5 17:26:37 2008 From: srichter at cosmos.phy.tufts.edu (Stephan Richter) Date: Sat Jul 5 17:26:32 2008 Subject: [Zope-dev] Zope 3 on Python 2.5, Zope 3 releases In-Reply-To: References: Message-ID: <200807051426.37527.srichter@cosmos.phy.tufts.edu> On Thursday 26 June 2008, Martijn Faassen wrote: > I'm curious about the plans of Zope 3 on Python 2.5. > > * Are people using Zope 3 with Python 2.5 already? What are your > experiences? Yes, works flawlessly. The problems in zope.security and zope.proxy that I recently found (and fixed) exist for Python 2.4 as well. > * Are there plans for a release after Zope 3.4? I'm just assuming that > Zope 3.4.0c1 is actually the only release that Zope 3.4 is going to see. Yeah, yeah. Next week I am traveling, so I plan to just update the packages to the latest bug-fix release and publish it. > Any plans or interest in a Zope 3.5? For me the interest is more of having a stable KGS than the TAR ball. > * Is someone exercising the "trunk" version of KGS, running the tests to > see whether everything works together? No I have not started doing that yet. > Is someone actually updating the versions? No, but producing it is not that much work. > Could we somehow broaden the community for this? Yes, please. It effectively means giving more people to the directories on download.zope.org. > What would the policy be for checkins of Zope-related packages? Could we > use new Python 2.5 features, for instance? That's a good question. I am +1 for that. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter" From benji at zope.com Sat Jul 5 20:28:07 2008 From: benji at zope.com (Benji York) Date: Sat Jul 5 20:27:54 2008 Subject: [Zope-dev] Parallel subprocesses ready to merge Message-ID: My branch that can (but by default doesn't) run subprocesses in parallel is ready to merge. It's not a panacea (if you have carefully crafted layers that can all be torn down, your test runs will probably be slower with the feature switched on), but for the common -- I believe -- case of having several layers that can't be torn down, tests run much faster on multi-core machines. The branch is svn+ssh://svn.zope.org/repos/main/zope.testing/branches/benji-parallelize-subprocesses. If no one finds major problems with the branch by the middle of next week, I'll merge it to the trunk. -- Benji York Senior Software Engineer Zope Corporation From gns24 at mythic-beasts.com Sat Jul 5 23:02:56 2008 From: gns24 at mythic-beasts.com (Graham Stratton) Date: Sat Jul 5 23:03:10 2008 Subject: [Zope-dev] Re: zc.testbrowser.real support for mozlab 0.1.9 In-Reply-To: References: <486CC2FF.6050906@gocept.com> <2A825211-BB78-4E77-932D-BFED09B877F7@mythic-beasts.com> Message-ID: On 4 Jul 2008, at 17:34, Christian Zagrodnick wrote: > On 2008-07-04 01:19:47 +0200, Graham Stratton beasts.com> said: >> I tried running the tests with Python 2.4. About half the time >> they pass fine, and the other half fail something like this: >> File "", line 1, in ? >> browser.open('navigate.html') >> File "/Users/graham/development/patched_zc.testbrowser/src/ >> zc/ testbrowser/real.py", line 201, in open >> self.wait() >> File "/Users/graham/development/patched_zc.testbrowser/src/ >> zc/ testbrowser/real.py", line 193, in wait >> assert self.execute('tb_page_loaded;') == 'false' >> AssertionError >> though for varying pages with browser.open(). I don't understand >> what could be causing it; why would that value be changed in >> between those two lines? I guess the time.sleep() on line 191 >> implies that there's something we might want to wait for. I >> increased it by 10x and only got 1 failure out of 10, so I guess >> that helps. > > I had this error, too and checked in a maybe-fix in r87964. Have you > tried with this revision or the one before? If it's thisone I'll add > another loop there. I've tried with this revision, and with the timeout extended, but still occasionally see the problem. I have a couple of independent fixes for other issues which I'll discuss once this branch is merged. Regards, Graham From zope-tests at epy.co.at Sun Jul 6 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Sun Jul 6 06:59:54 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807061100.m66B05sX022585@shnoll.test.at> Summary of messages to the zope-tests list. Period Sat Jul 5 11:00:00 2008 UTC to Sun Jul 6 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Sat Jul 5 21:02:28 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009810.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Sat Jul 5 21:03:58 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009811.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Sat Jul 5 21:05:28 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009812.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Sat Jul 5 21:06:58 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009813.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Sat Jul 5 21:08:28 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009814.html From zope-tests at epy.co.at Mon Jul 7 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Mon Jul 7 06:59:53 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807071100.m67B05Sp009840@shnoll.test.at> Summary of messages to the zope-tests list. Period Sun Jul 6 11:00:00 2008 UTC to Mon Jul 7 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Sun Jul 6 21:08:10 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009815.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Sun Jul 6 21:09:40 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009816.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Sun Jul 6 21:11:10 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009817.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Sun Jul 6 21:12:40 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009818.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Sun Jul 6 21:14:10 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009819.html From tim at sitefusion.co.uk Mon Jul 7 16:21:42 2008 From: tim at sitefusion.co.uk (Tim Hicks) Date: Mon Jul 7 16:21:43 2008 Subject: [Zope-dev] Product version number redundancy with version.txt and setup.py Message-ID: <48727AD6.2090901@sitefusion.co.uk> Hi, Am I right in thinking that there is a duplication of information in having an eggified product's version number stored in setup.py and version.txt? Both seem to be necessary - the first for use with pypi and the second so that Zope knows what version of a product it has. The former seems more like it should be the preferred single place for this information. Does anyone have any ideas about how we might update Zope to read version information from setup.py instead? The initializeProduct function at seems to be the relevant place. However, I'm not sure if that code has access to the full eggy package at that stage. My guess is that the relevant object (productp?) might already have been adjusted for backward compatibility reasons at that stage. Any ideas? Tim From plone at hannosch.info Mon Jul 7 16:39:16 2008 From: plone at hannosch.info (Hanno Schlichting) Date: Mon Jul 7 16:39:15 2008 Subject: [Zope-dev] Re: Product version number redundancy with version.txt and setup.py In-Reply-To: <48727AD6.2090901@sitefusion.co.uk> References: <48727AD6.2090901@sitefusion.co.uk> Message-ID: <48727EF4.6080907@hannosch.info> Tim Hicks wrote: > Am I right in thinking that there is a duplication of information in > having an eggified product's version number stored in setup.py and > version.txt? Having your setup.py read its version from version.txt is easy, if you don't want to update two places. I think the more interesting question is, why we need a persistent registry of installed products and any information about them at all anymore? Having the product registry persistent causes quite some trouble when moving Data.fs across servers and doesn't seem to add much to any benefit. For what kind of use-case do people use that registry nowadays, since we got rid of persistent products? Hanno From wichert at wiggy.net Mon Jul 7 16:44:43 2008 From: wichert at wiggy.net (Wichert Akkerman) Date: Mon Jul 7 16:44:30 2008 Subject: [Zope-dev] Re: Product version number redundancy with version.txt and setup.py In-Reply-To: <48727EF4.6080907@hannosch.info> References: <48727AD6.2090901@sitefusion.co.uk> <48727EF4.6080907@hannosch.info> Message-ID: <20080707204443.GB28336@wiggy.net> Previously Hanno Schlichting wrote: > Tim Hicks wrote: > >Am I right in thinking that there is a duplication of information in > >having an eggified product's version number stored in setup.py and > >version.txt? > > Having your setup.py read its version from version.txt is easy, if you > don't want to update two places. > > I think the more interesting question is, why we need a persistent > registry of installed products and any information about them at all > anymore? ZClasses Wichert. -- Wichert Akkerman It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. From limi at plone.org Tue Jul 8 05:49:30 2008 From: limi at plone.org (Alexander Limi) Date: Tue Jul 8 05:49:29 2008 Subject: [Zope-dev] Re: Product version number redundancy with version.txt and setup.py References: <48727AD6.2090901@sitefusion.co.uk> <48727EF4.6080907@hannosch.info> <20080707204443.GB28336@wiggy.net> Message-ID: On Mon, 07 Jul 2008 13:44:43 -0700, Wichert Akkerman wrote: > Previously Hanno Schlichting wrote: >> Tim Hicks wrote: >> >Am I right in thinking that there is a duplication of information in >> >having an eggified product's version number stored in setup.py and >> >version.txt? >> >> Having your setup.py read its version from version.txt is easy, if you >> don't want to update two places. >> >> I think the more interesting question is, why we need a persistent >> registry of installed products and any information about them at all >> anymore? > > ZClasses Speaking of which, are there plans to rip out ZClasses from Zope 2.12? It's about time! -- Alexander Limi ? http://limi.net From zope-tests at epy.co.at Tue Jul 8 07:00:06 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Tue Jul 8 06:59:53 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807081100.m68B06hm025147@shnoll.test.at> Summary of messages to the zope-tests list. Period Mon Jul 7 11:00:00 2008 UTC to Tue Jul 8 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Mon Jul 7 21:01:10 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009820.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Mon Jul 7 21:02:40 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009821.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Mon Jul 7 21:04:10 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009822.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Mon Jul 7 21:05:40 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009823.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Mon Jul 7 21:07:10 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009824.html From wichert at wiggy.net Tue Jul 8 08:13:35 2008 From: wichert at wiggy.net (Wichert Akkerman) Date: Tue Jul 8 08:13:22 2008 Subject: [Zope-dev] zope.testing releases missing on pypi Message-ID: <20080708121335.GA9910@wiggy.net> I see zope.testing 3.5.2 and 3.5.3 in svn and used by Zope2 but the latest version on pypi is 3.5.1. Why is that? And can that be corrected? Wichert. -- Wichert Akkerman It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. From lists at zopyx.com Tue Jul 8 09:07:20 2008 From: lists at zopyx.com (Andreas Jung) Date: Tue Jul 8 09:07:09 2008 Subject: [Zope-dev] Re: zope.testing releases missing on pypi In-Reply-To: <20080708121335.GA9910@wiggy.net> References: <20080708121335.GA9910@wiggy.net> Message-ID: <5733A531A23046FC662D7535@[192.168.0.24]> In fact I uploaded zope.testing 3.5.2 on Sunday to PyPI. However I also noticed that 3.5.2 disappeared this morning and that I had no longer permissions on zope.testing...no idea what's going on. Andreas --On 8. Juli 2008 14:13:35 +0200 Wichert Akkerman wrote: > I see zope.testing 3.5.2 and 3.5.3 in svn and used by Zope2 but the > latest version on pypi is 3.5.1. Why is that? And can that be corrected? > > Wichert. > > -- > Wichert Akkerman It is simple to make things. > http://www.wiggy.net/ It is hard to make things simple. -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany Web: www.zopyx.com - Email: info@zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080708/ea96d112/attachment.bin From dev at projekt01.ch Tue Jul 8 11:41:15 2008 From: dev at projekt01.ch (Roger Ineichen) Date: Tue Jul 8 11:40:36 2008 Subject: [Zope-dev] zope.app.authentication broken! Message-ID: hi I'm pretty sure the container changes are not compatible because of some bad __init__ methods in inherited classes in other packages. But that's not the fault of the refactoring that is correct as far as I can see. zope.app.authentication.groupfolder.py class GroupFolder(BTreeContainer): .... def __init__(self, prefix=u''): self.prefix=prefix super(BTreeContainer,self).__init__() ^^^^^^^^^^^^^^^^^^^^ The *super(BTreeContainer,self).__init__()* will not initialize the right thing here. right? What was the reason of calling: super(BTreeContainer,self).__init__() instead of: super(GroupFolder, self).__init__() btw, did someone run the tests with all packages that depend on zope.app.container? Regards Roger Ineichen _____________________________ END OF MESSAGE From philipp at weitershausen.de Tue Jul 8 14:55:00 2008 From: philipp at weitershausen.de (Philipp von Weitershausen) Date: Tue Jul 8 14:54:59 2008 Subject: [Zope-dev] Re: SVN: zope.app.container/trunk/CHANGES.txt Added release note In-Reply-To: <20080708174650.B849839988@mail.zope.org> References: <20080708174650.B849839988@mail.zope.org> Message-ID: <4873B804.4010103@weitershausen.de> Roger Ineichen wrote: > Log message for revision 88121: > Added release note > > Changed: > U zope.app.container/trunk/CHANGES.txt > > -=- > Modified: zope.app.container/trunk/CHANGES.txt > =================================================================== > --- zope.app.container/trunk/CHANGES.txt 2008-07-08 17:45:53 UTC (rev 88120) > +++ zope.app.container/trunk/CHANGES.txt 2008-07-08 17:46:50 UTC (rev 88121) > @@ -2,6 +2,13 @@ > CHANGES > ======= > > + > +Heads up: > + > +Relase zope.app.authentication 3.4.2 if this 3.6.1 get released. It contains a > +bugfix based on the BTreeContainer re-implementation. > + > + I think this it can be released now. The fix you applied was needed either way, if I'm correct. GroupFolder's __init__ was not calling its own super's __init__ but the super __init__ of BTreeFolder. This should have worked before and after the changes in zope.app.container 3.6.x. Of course, it'd be good to test this with both zope.app.container 3.5.x and 3.6.x first :) From dev at projekt01.ch Wed Jul 9 05:14:20 2008 From: dev at projekt01.ch (Roger Ineichen) Date: Wed Jul 9 05:13:41 2008 Subject: [Zope-dev] AW: SVN: zope.app.container/trunk/CHANGES.txt Added release note In-Reply-To: <4873B804.4010103@weitershausen.de> References: <20080708174650.B849839988@mail.zope.org> <4873B804.4010103@weitershausen.de> Message-ID: Hi Philipp > Betreff: Re: SVN: zope.app.container/trunk/CHANGES.txt Added > release note [...] > I think this it can be released now. The fix you applied was > needed either way, if I'm correct. GroupFolder's __init__ was > not calling its own super's __init__ but the super __init__ > of BTreeFolder. This should have worked before and after the > changes in zope.app.container 3.6.x. Yes > Of course, it'd be good to test this with both > zope.app.container 3.5.x and 3.6.x first :) It works on my old and new projects which uses different container packages. Just released zope.app.authentication 3.4.2 Regards Roger Ineichen _____________________________ END OF MESSAGE From dusty at qwer.tk Wed Jul 9 05:14:48 2008 From: dusty at qwer.tk (Hermann Himmelbauer) Date: Wed Jul 9 05:14:53 2008 Subject: [Zope-dev] Zope3 installation on windows - how? Message-ID: <200807091114.49227.dusty@qwer.tk> Hi, I currently try to install Zope3-4.0c1 on a windows machine, without any success for now. What I need is an installation that includes basically all packages that are in the tarball into the python path. I tried the following: 1) Tarball: Here, install.py fails due to some compiler problems - it suggests to specify "-c mingw32", however, when I do so, install.py complains that it does not know this "-c mingw32" option. Something like "python install.py build_ext -cmingw32" works, it seems that all needed packages are correctly compiled. When I try a "python install.py install" afterwards, it once again complains that it needs mingw32. So I'm stuck. 2) I tried installing it via the descriptions given here: http://wiki.zope.org/zope3/Zope340c1 $ svn co svn://svn.zope.org/repos/main/z3c.formdemo/tags/1.5.1 formdemo $ cd formdemo $ python bootstrap.py $ ./bin/buildout -v This gives me some buildout error. 3) Next I tried the information from here: http://wiki.zope.org/zope3/MakingARelease svn co svn+ssh://svn.zope.org/repos/main/zope.release/branches/3.4 release-3.4 $ cd release-3.4 $ py24 bootstrap.py $ ./bin/buildout -N And there's once again this mingw32 problem. 4) Next, I tried zopeproject: $ easy_install zopeproject $ zopeproject HelloWorld First, zopeproject does not install the packages into the Python site-packages, moreover, there's once again the "mingw32" problem which I'm unable to solve. What can I do? Is there perhaps some precompiled Windows distribution for Zope-3.4 available? Or is there some trick to make things work with mingw32? Best Regards, Hermann -- hermann@qwer.tk GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7 From zope-tests at epy.co.at Wed Jul 9 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Wed Jul 9 06:59:53 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807091100.m69B05fN009398@shnoll.test.at> Summary of messages to the zope-tests list. Period Tue Jul 8 11:00:00 2008 UTC to Wed Jul 9 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Tue Jul 8 21:09:04 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009825.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 8 21:10:35 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009826.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 8 21:12:05 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009827.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 8 21:13:35 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009828.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Tue Jul 8 21:15:05 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009829.html From dusty at qwer.tk Wed Jul 9 07:54:48 2008 From: dusty at qwer.tk (Hermann Himmelbauer) Date: Wed Jul 9 07:54:38 2008 Subject: [Zope-dev] Zope3 installation on windows - how? In-Reply-To: <200807091114.49227.dusty@qwer.tk> References: <200807091114.49227.dusty@qwer.tk> Message-ID: <200807091354.48521.dusty@qwer.tk> Am Mittwoch, 9. Juli 2008 11:14 schrieb Hermann Himmelbauer: > Hi, > I currently try to install Zope3-4.0c1 on a windows machine, without any > success for now. What I need is an installation that includes basically > all packages that are in the tarball into the python path. I tried the > following: > > 1) Tarball: Here, install.py fails due to some compiler problems - it > suggests to specify "-c mingw32", however, when I do so, install.py > complains that it does not know this "-c mingw32" option. Something like > "python install.py build_ext -cmingw32" works, it seems that all needed > packages are correctly compiled. When I try a "python install.py install" > afterwards, it once again complains that it needs mingw32. So I'm stuck. Ok, thanks to a helpful hint, I was able to solve it. One has to create a file "pydistutils.py" in a directory, which is specified in an environment variable "HOME" with the following content: [build] compiler=mingw32 Then the "python install.py install" in the tarball worked. Nevertheless, it would be optimal, if "install.py" is adapted in a way so that "-c mingw32" works... Regards, Hermann -- hermann@qwer.tk GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7 From vincent at nexedi.com Wed Jul 9 09:00:32 2008 From: vincent at nexedi.com (Vincent Pelletier) Date: Wed Jul 9 09:03:02 2008 Subject: [Zope-dev] Dangerous shutdown procedure behaviour Message-ID: <200807091500.33603.vincent@nexedi.com> Hi. I think I discovered a dangerous code behaviour at zope shutdown. I've had a strange problem on a site where persistent objects are created from data inserted in an SQL table. Upon object creation, SQL table is updated to mark the line as imported. Such import got triggered just before a shutdown. After restart and another import, documents were created twice. What I believe happened (but I could not find any hard evidence of it) is that Zope blindly exited while the working thread was runing, and in the worst possible method: tpc_finish. ZODB was already commited, but mysql was not. So mysql did a rollback on changes, and the lines were in a "ready to import" state. And imported again at next import attemp. Reading shutdown code, I discovered 2 distinct timeout mechanism (note: having just one is enough to trigger the problem): - Lifetime.py: iterating through asyncore sockets, it alerts servers that it will shut down soon. If they take the veto for too long, the veto is ignored and shutdown continues. Default timeout is 20 seconds, meaning there is at most one minute from the first shutdown notice to the effective process exit (taking all runing threads down). When invoking "zopectl stop", it's runing a "fast" shutdown, which means the timeout is shortened to 1 second, so total maximum sutdown time is 3 seconds. This timeout can be worked around by just writing blocking shutdown methods and not using the veto system. - zdaemon/zdrun.py: if the instance being shut down still responds after 10 seconds, it will be sent a SIGKILL. This cannot be worked around without changing code in zdrun.py or not executing it at all (no idea if there is any alternative). I could easily reproduce the problem by writing a simple connection mamager which calls time.wait(3600) in _finish method and defining a sortKey method to make it commit after another connection manager. I could not find a trace of any mechanism preventing commit from happening when a shutdown is in progress, and I don't think there should be any: considering that some storages might be accessed through a network, latency can become a problem, so tpc_finish can take time to complete, so just checking that there is no pending shutdown before entering this function would not solve the problem. I suggest removing all those timeouts. If a user wants a Zope to shutdown for a reason serious enough to send it a SIGKILL or causing immediate python thread termination, it's his responsibility. But I think regular shutdown mechanism must not do that. Also, the same problem can happen with "zopectl fg" since Zope does not go through any shutdown sequence as far as I can tell (it just dies). -- Vincent Pelletier From lists at zopyx.com Wed Jul 9 23:37:48 2008 From: lists at zopyx.com (Andreas Jung) Date: Wed Jul 9 23:37:46 2008 Subject: [Zope-dev] zope.testing releases on PyPI Message-ID: <2E006C07DE4575D0F9F3573D@lap80031.grp.haufemg.com> Hi, the zope.testing releases on PyPI are apparently a bit weird. - zope.testing 3.5.1 has been the latest released version on PyPI last week - I created V 3.5.2 on Sunday. Phillip granted me access to the zope.testing package on PyPI in order to upload 3.5.2 (which worked) - I created V 3.5.3 yesterday morning. Uploading the egg was impossible because I had no longer rights (in fact permissions are currentl only granted to Jim and Fred Drake (not even Phillip) - search for "zope.testing" on PyPI leads me to version 3.0 instead of 3.5.1 Does anyone know what happened here? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080710/df2cc8cb/attachment.bin From slinkp at gmail.com Thu Jul 10 00:12:06 2008 From: slinkp at gmail.com (Paul Winkler) Date: Thu Jul 10 00:11:53 2008 Subject: [Zope-dev] transaction.doom() and ZPublisher Message-ID: <20080710041205.GA4449@slinkp.com> Hi, I noticed that Zope 2.11 includes a recent version of the transaction module including the transaction.doom() method. But I don't see any check for it in ZPublisher. So, if any code calls transaction.doom(), the publisher will raise a user-visible exception when it tries to call commit(). This seems less than useful :-) In the discussion I've seen of this method, eg. https://bugs.launchpad.net/zope3/+bug/98382 and http://markmail.org/message/3yshpmltvhevnrff it sounds like other people share my expectation... namely, that the developer should not have to do anything special after calling transaction.doom(); the transaction is not committed, and the user won't see an exception. Is that the concensus? If so, there's an easy solution - apply something like the attached patch. Anybody disagree? -- Paul Winkler http://www.slinkp.com -------------- next part -------------- --- Zope2/App/startup.py~ 2008-06-14 02:50:23.000000000 -0400 +++ Zope2/App/startup.py 2008-07-10 00:08:02.000000000 -0400 @@ -267,7 +267,8 @@ transaction.begin() def commit(self): - transaction.commit() + if not transaction.isDoomed(): + transaction.commit() def abort(self): transaction.abort() --- ZPublisher/Publish.py~ 2008-06-14 02:50:48.000000000 -0400 +++ ZPublisher/Publish.py 2008-07-10 00:08:08.000000000 -0400 @@ -314,7 +314,8 @@ def begin(self): transaction.begin() def commit(self): - transaction.commit() + if not transaction.get().isDoomed(): + transaction.commit() def abort(self): transaction.abort() def recordMetaData(self, object, request): --- ZPublisher/WSGIPublisher.py~ 2008-06-14 02:50:48.000000000 -0400 +++ ZPublisher/WSGIPublisher.py 2008-07-09 23:59:57.000000000 -0400 @@ -401,7 +401,8 @@ def begin(self): transaction.begin() def commit(self): - transaction.commit() + if not transaction.get().isDoomed(): + transaction.commit() def abort(self): transaction.abort() def recordMetaData(self, object, request): From zope-tests at epy.co.at Thu Jul 10 07:00:05 2008 From: zope-tests at epy.co.at (Zope Tests Summarizer) Date: Thu Jul 10 06:59:52 2008 Subject: [Zope-dev] Zope Tests: 5 OK Message-ID: <200807101100.m6AB05p3010698@shnoll.test.at> Summary of messages to the zope-tests list. Period Wed Jul 9 11:00:00 2008 UTC to Thu Jul 10 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --------------- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Wed Jul 9 21:00:51 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009830.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 9 21:02:21 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009831.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 9 21:03:51 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009832.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 9 21:05:22 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009833.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Wed Jul 9 21:06:52 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-July/009834.html From ccomb at free.fr Thu Jul 10 08:15:16 2008 From: ccomb at free.fr (Christophe Combelles) Date: Thu Jul 10 08:14:51 2008 Subject: [Zope-dev] zope.app.authentication broken! In-Reply-To: References: Message-ID: <4875FD54.40201@free.fr> Roger Ineichen a ?crit : > hi > > I'm pretty sure the container changes are not compatible because of > some bad __init__ methods in inherited classes in other packages. > > But that's not the fault of the refactoring that is correct > as far as I can see. > > zope.app.authentication.groupfolder.py > > class GroupFolder(BTreeContainer): > > .... > > def __init__(self, prefix=u''): > self.prefix=prefix > super(BTreeContainer,self).__init__() > ^^^^^^^^^^^^^^^^^^^^ > > > The *super(BTreeContainer,self).__init__()* will > not initialize the right thing here. right? > > What was the reason of calling: > super(BTreeContainer,self).__init__() > instead of: > super(GroupFolder, self).__init__() > > btw, did someone run the tests with all packages that depend > on zope.app.container? I would like to do this. How can I know all the packages that depend on zope.app.container? What is the procedure? Then how can I run all the tests? Christophe > > Regards > Roger Ineichen > _____________________________ > END OF MESSAGE > > _______________________________________________ > Zope-Dev maillist - Zope-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) > > From ccomb at free.fr Thu Jul 10 08:31:31 2008 From: ccomb at free.fr (Christophe Combelles) Date: Thu Jul 10 08:31:05 2008 Subject: [Zope-dev] zope.app.authentication broken! In-Reply-To: <4875FD54.40201@free.fr> References: <4875FD54.40201@free.fr> Message-ID: <48760123.3030208@free.fr> Christophe Combelles a ?crit : > Roger Ineichen a ?crit : >> hi >> >> I'm pretty sure the container changes are not compatible because of >> some bad __init__ methods in inherited classes in other packages. >> >> But that's not the fault of the refactoring that is correct as far as >> I can see. >> >> zope.app.authentication.groupfolder.py >> >> class GroupFolder(BTreeContainer): >> >> .... >> >> def __init__(self, prefix=u''): >> self.prefix=prefix >> super(BTreeContainer,self).__init__() >> ^^^^^^^^^^^^^^^^^^^^ >> >> >> The *super(BTreeContainer,self).__init__()* will >> not initialize the right thing here. right? >> >> What was the reason of calling: >> super(BTreeContainer,self).__init__() >> instead of: >> super(GroupFolder, self).__init__() >> >> btw, did someone run the tests with all packages that depend >> on zope.app.container? > > I would like to do this. How can I know all the packages that depend on > zope.app.container? What is the procedure? Then how can I run all the > tests? ok, it's explained in zope.kgs > > Christophe > > >> >> Regards >> Roger Ineichen >> _____________________________ >> END OF MESSAGE >> >> _______________________________________________ >> Zope-Dev maillist - Zope-Dev@zope.org >> http://mail.zope.org/mailman/listinfo/zope-dev >> ** No cross posts or HTML encoding! ** >> (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce >> http://mail.zope.org/mailman/listinfo/zope ) >> >> > > _______________________________________________ > Zope-Dev maillist - Zope-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) > > From dev at projekt01.ch Thu Jul 10 08:47:36 2008 From: dev at projekt01.ch (Roger Ineichen) Date: Thu Jul 10 08:47:00 2008 Subject: AW: [Zope-dev] zope.app.authentication broken! In-Reply-To: <48760123.3030208@free.fr> References: <4875FD54.40201@free.fr> <48760123.3030208@free.fr> Message-ID: <6CD63868A03E4E49A1BB2ABCE60B2EA4@mobile04> Hi Chritophe > Betreff: Re: [Zope-dev] zope.app.authentication broken! [...] > >> btw, did someone run the tests with all packages that depend on > >> zope.app.container? > > > > I would like to do this. How can I know all the packages > that depend > > on zope.app.container? What is the procedure? Then how can > I run all > > the tests? > > > ok, it's explained in zope.kgs Yeah, thanks! I already released the zope.app.authentication package fix. Regards Roger Ineichen > > Christophe From martijn at fourdigits.nl Thu Jul 10 09:06:53 2008 From: martijn at fourdigits.nl (Martijn Jacobs) Date: Thu Jul 10 09:06:43 2008 Subject: [Zope-dev] ZCatalog sorting issue Message-ID: <4876096D.3060404@fourdigits.nl> Wat denk je hiervan? Hello. In zope 2.10.5 (and probably 2.10.6 and 2.11 and, as I've read, all releases above 2.7) we've encountered a sorting bug in a dtml-in call when querying the catalog. I don't think it's dtml only related, but I'm not sure about that. It is the same bug as found on : https://bugs.launchpad.net/zope2/+bug/143504 Adding def __cmp__(self, other): return 0 to Products.ZCatalog.CatalogBrains.AbstractCatalogBrain seems to correct the problem though and I was wondering why this isn't added in the zope core. Does it break other stuff or should the problem fixed somewhere else? Somebody has some thoughts? Regards, Martijn -- Martijn Jacobs Four Digits, Internet Solutions a: Willemsplein 15-1 6811 KB Arnhem NL kvk: 091621370000 | btw: 8161.22.234.B01 e-mail: martijn@fourdigits.nl | web: http://www.fourdigits.nl tel: +31 (0)26 44 22 700 | fax: +31 (0)84 22 06 117 From jim at zope.com Thu Jul 10 09:09:40 2008 From: jim at zope.com (Jim Fulton) Date: Thu Jul 10 09:09:29 2008 Subject: [Zope-dev] Proposal: add form support to zope.session.http.CookieClientIdManager Message-ID: <336C24D0-C331-4FA1-B2EC-86C4D4229030@zope.com> As it's name implies, the CookieClientIdManager manages HTTP client ids using HTTP cookies. Recently, I've found it to be useful to be able to get a client id via form data. The reason is that flash programs don't send browser cookies. This bit me when trying to use the YUI uploader, http://developer.yahoo.com/yui/uploader/, which uses a flash program to select and upload files. I'm using session authentication and the only way I found to make it possible to upload files to protected pages was to pass the client id as form data. I have a custom client id manager that gets the id from form data as well as from a cookie. I propose to merge this with CookieClientIdManager. Any objections? Jim -- Jim Fulton Zope Corporation From dev at projekt01.ch Thu Jul 10 09:28:48 2008 From: dev at projekt01.ch (Roger Ineichen) Date: Thu Jul 10 09:28:11 2008 Subject: AW: [Zope-dev] Proposal: add form support tozope.session.http.CookieClientIdManager In-Reply-To: <336C24D0-C331-4FA1-B2EC-86C4D4229030@zope.com> References: <336C24D0-C331-4FA1-B2EC-86C4D4229030@zope.com> Message-ID: <0F8CCD4A451D482DB8D105E702263FE7@mobile04> Hi Jim > Betreff: [Zope-dev] Proposal: add form support > tozope.session.http.CookieClientIdManager > > > As it's name implies, the CookieClientIdManager manages HTTP > client ids using HTTP cookies. Recently, I've found it to be > useful to be able to get a client id via form data. The > reason is that flash programs don't send browser cookies. > This bit me when trying to use the YUI uploader, > http://developer.yahoo.com/yui/uploader/, which uses a flash > program to select and upload files. I'm using session > authentication and the only way I found to make it possible > to upload files to protected pages was to pass the client id > as form data. I have a custom client id manager that gets > the id from form data as well as from a cookie. I propose to > merge this with CookieClientIdManager. > > Any objections? Does this bring in new dependencies to zope.session? if not moreDependencies: +1 else: -1 btw, I hope it will be still possible to remove the zope.app.appsetup somedays from the zope.session package. Regards Roger Ineichen > Jim > > -- > Jim Fulton > Zope Corporation From jim at zope.com Thu Jul 10 09:42:36 2008 From: jim at zope.com (Jim Fulton) Date: Thu Jul 10 09:42:26 2008 Subject: [Zope-dev] zope.testing releases on PyPI In-Reply-To: <2E006C07DE4575D0F9F3573D@lap80031.grp.haufemg.com> References: <2E006C07DE4575D0F9F3573D@lap80031.grp.haufemg.com> Message-ID: On Jul 9, 2008, at 11:37 PM, Andreas Jung wrote: > > Hi, > > the zope.testing releases on PyPI are apparently a bit weird. > > - zope.testing 3.5.1 has been the latest released version on PyPI > last week > - I created V 3.5.2 on Sunday. Phillip granted me access to the > zope.testing package on PyPI in order to upload 3.5.2 (which worked) > - I created V 3.5.3 yesterday morning. Uploading the egg was > impossible > because I had no longer rights (in fact permissions are currentl > only granted to Jim and Fred Drake (not even Phillip) > - search for "zope.testing" on PyPI leads me to version 3.0 instead > of 3.5.1 > > Does anyone know what happened here? Of course not. :) I suggest asking the catalog-sig. I suspect someone restored the db from backup. That or database corruption of some sort. Relational DBs are soooo unreliable. ;) Jim -- Jim Fulton Zope Corporation From jim at zope.com Thu Jul 10 09:45:19 2008 From: jim at zope.com (Jim Fulton) Date: Thu Jul 10 09:45:12 2008 Subject: AW: [Zope-dev] Proposal: add form support tozope.session.http.CookieClientIdManager In-Reply-To: <0F8CCD4A451D482DB8D105E702263FE7@mobile04> References: <336C24D0-C331-4FA1-B2EC-86C4D4229030@zope.com> <0F8CCD4A451D482DB8D105E702263FE7@mobile04> Message-ID: On Jul 10, 2008, at 9:28 AM, Roger Ineichen wrote: > Hi Jim > >> Betreff: [Zope-dev] Proposal: add form support >> tozope.session.http.CookieClientIdManager >> >> >> As it's name implies, the CookieClientIdManager manages HTTP >> client ids using HTTP cookies. Recently, I've found it to be >> useful to be able to get a client id via form data. The >> reason is that flash programs don't send browser cookies. >> This bit me when trying to use the YUI uploader, >> http://developer.yahoo.com/yui/uploader/, which uses a flash >> program to select and upload files. I'm using session >> authentication and the only way I found to make it possible >> to upload files to protected pages was to pass the client id >> as form data. I have a custom client id manager that gets >> the id from form data as well as from a cookie. I propose to >> merge this with CookieClientIdManager. >> >> Any objections? > > Does this bring in new dependencies to zope.session? No. Jim -- Jim Fulton Zope Corporation From srichter at cosmos.phy.tufts.edu Wed Jul 9 23:49:49 2008 From: srichter at cosmos.phy.tufts.edu (Stephan Richter) Date: Thu Jul 10 10:41:08 2008 Subject: [Zope-dev] zope.testing releases on PyPI In-Reply-To: <2E006C07DE4575D0F9F3573D@lap80031.grp.haufemg.com> References: <2E006C07DE4575D0F9F3573D@lap80031.grp.haufemg.com> Message-ID: <200807092049.49403.srichter@cosmos.phy.tufts.edu> On Wednesday 09 July 2008, Andreas Jung wrote: > ?- search for "zope.testing" on PyPI leads me to version 3.0 instead of > 3.5.1 I think this is because all releases are "visible". And it probably picks the first visible one and not the last. I do not have access zope.testing to verify that theory. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter" From brian at vanguardistas.net Thu Jul 10 11:09:36 2008 From: brian at vanguardistas.net (Brian Sutherland) Date: Thu Jul 10 11:09:29 2008 Subject: [Zope-dev] transaction.doom() and ZPublisher In-Reply-To: <20080710041205.GA4449@slinkp.com> References: <20080710041205.GA4449@slinkp.com> Message-ID: <20080710150936.GD4818@brian-sutherlands-macbook.local> On Thu, Jul 10, 2008 at 12:12:06AM -0400, Paul Winkler wrote: > Hi, > > I noticed that Zope 2.11 includes a recent version of the transaction > module including the transaction.doom() method. But I don't see any > check for it in ZPublisher. > > So, if any code calls transaction.doom(), the publisher will raise a > user-visible exception when it tries to call commit(). This seems > less than useful :-) > > In the discussion I've seen of this method, eg. > https://bugs.launchpad.net/zope3/+bug/98382 and > http://markmail.org/message/3yshpmltvhevnrff it sounds like other > people share my expectation... namely, that the developer should not > have to do anything special after calling transaction.doom(); the > transaction is not committed, and the user won't see an exception. Yes. > > Is that the concensus? If so, there's an easy solution - > apply something like the attached patch. > > Anybody disagree? I havn't investigated properly, but it may be necessary to do the isDoomed() check at a higher level where you can abort. Code running after the commit() expects a new transaction and now will not get that. Other than that, an explicit abort() if the transaction is doomed may be sufficient. Zope3 does: def afterCall(self, request, ob): txn = transaction.get() if txn.isDoomed(): txn.abort() else: self.annotateTransaction(txn, request, ob) txn.commit() > > > -- > > Paul Winkler > http://www.slinkp.com > --- Zope2/App/startup.py~ 2008-06-14 02:50:23.000000000 -0400 > +++ Zope2/App/startup.py 2008-07-10 00:08:02.000000000 -0400 > @@ -267,7 +267,8 @@ > transaction.begin() > > def commit(self): > - transaction.commit() > + if not transaction.isDoomed(): > + transaction.commit() > > def abort(self): > transaction.abort() > --- ZPublisher/Publish.py~ 2008-06-14 02:50:48.000000000 -0400 > +++ ZPublisher/Publish.py 2008-07-10 00:08:08.000000000 -0400 > @@ -314,7 +314,8 @@ > def begin(self): > transaction.begin() > def commit(self): > - transaction.commit() > + if not transaction.get().isDoomed(): > + transaction.commit() > def abort(self): > transaction.abort() > def recordMetaData(self, object, request): > --- ZPublisher/WSGIPublisher.py~ 2008-06-14 02:50:48.000000000 -0400 > +++ ZPublisher/WSGIPublisher.py 2008-07-09 23:59:57.000000000 -0400 > @@ -401,7 +401,8 @@ > def begin(self): > transaction.begin() > def commit(self): > - transaction.commit() > + if not transaction.get().isDoomed(): > + transaction.commit() > def abort(self): > transaction.abort() > def recordMetaData(self, object, request): > _______________________________________________ > Zope-Dev maillist - Zope-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) -- Brian Sutherland From sidnei at enfoldsystems.com Thu Jul 10 13:38:20 2008 From: sidnei at enfoldsystems.com (Sidnei da Silva) Date: Thu Jul 10 13:38:38 2008 Subject: [Zope-dev] Re: Deprecation warnings from 2.11.0 In-Reply-To: <486E681F.1040505@simplistix.co.uk> References: <891A9FDE8A80387848366BBA@192.168.0.24> <4860BD8A.6030104@simplistix.co.uk> <48620149.5080702@simplistix.co.uk> <486E681F.1040505@simplistix.co.uk> Message-ID: On Fri, Jul 4, 2008 at 3:12 PM, Chris Withers wrote: > Did we as a community really release a stable version that emits deprecation > warnings?! :-( We did (/me hides). -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 From wichert at wiggy.net Thu Jul 10 13:50:15 2008 From: wichert at wiggy.net (Wichert Akkerman) Date: Thu Jul 10 13:50:32 2008 Subject: [Zope-dev] zope.testing releases on PyPI In-Reply-To: <200807092049.49403.srichter@cosmos.phy.tufts.edu> References: <2E006C07DE4575D0F9F3573D@lap80031.grp.haufemg.com> <200807092049.49403.srichter@cosmos.phy.tufts.edu> Message-ID: <20080710175015.GA9719@wiggy.net> Previously Stephan Richter wrote: > On Wednesday 09 July 2008, Andreas Jung wrote: > > ?- search for "zope.testing" on PyPI leads me to version 3.0 instead of > > 3.5.1 > > I think this is because all releases are "visible". And it probably picks the > first visible one and not the last. I do not have access zope.testing to > verify that theory. Pypi always marks the last upload as the only visible version, which means Andreas's two uploads should have been visible. A quick peek at http://pypi.python.org/simple/zope.testing/ shows that 3.5.2 really was removed. Wichert. -- Wichert Akkerman It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. From ranjithkannikara at gmail.com Thu Jul 10 13:45:42 2008 From: ranjithkannikara at gmail.com (ranjith kannikara) Date: Thu Jul 10 13:55:59 2008 Subject: [Zope-dev] RestrictedPython implementation in zope2. Message-ID: <20aa8c370807101045w608cdafepf9c87fca0659e3ae@mail.gmail.com> Hello, During the porting of zope2 to python2.5 I am in need and guidance on doing the security auditing of RestrictedPython for python2.5 . Now a person named Chris Withers had volunteered for helping. And I will be happy to get guidance and help from Chris Withers. From philipp at weitershausen.de Thu Jul 10 16:04:21 2008 From: philipp at weitershausen.de (Philipp von Weitershausen) Date: Thu Jul 10 16:04:26 2008 Subject: [Zope-dev] Re: Zope3 installation on windows - how? In-Reply-To: <200807091114.49227.dusty@qwer.tk> References: <200807091114.49227.dusty@qwer.tk> Message-ID: Hermann Himmelbauer wrote: > And there's once again this mingw32 problem. > > 4) Next, I tried zopeproject: > $ easy_install zopeproject > $ zopeproject HelloWorld > > First, zopeproject does not install the packages into the Python > site-packages, That's *intended*. That's, in fact, much better than stuffing about a 100 eggs into site-packages... > What can I do? Is there perhaps some precompiled Windows distribution for > Zope-3.4 available? All Zope 3.4 packages that have C extensions are available as Windows binaries, though perhaps not in all versions. The versions in the KGS should all be available as Windows binaries, though. Just include http://download.zope.org/zope3.4/versions.cfg in your buildout.cfg. From flo at chaoflow.net Thu Jul 10 16:07:35 2008 From: flo at chaoflow.net (Florian Friesdorf) Date: Thu Jul 10 16:16:21 2008 Subject: [Zope-dev] PAULA: bringing Zope 3's authentication to Plone and beyond Message-ID: <20080710200735.GA20161@marvin> Hi *, within the scope of google summer of code I am integrating zope 3's PAU with Plone's PAS and further enable (non-AT) content objects as source for users and groups. All functionality is developed in pure zope3, the plone integration is happening in a separate packages. All documents describing the project, as well as links to the code can be found here: https://chaoflow.net/projects/gsoc2008/z3membrane-ldap The code is not very interesting right now, but now would be the time to take any influence on what will be created during the next month - I am planning to continue to work on the project after the SoC. I will keep you updated on major advancements of the code. regards florian -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080710/cc976848/attachment.bin From flo at chaoflow.net Thu Jul 10 16:22:17 2008 From: flo at chaoflow.net (Florian Friesdorf) Date: Thu Jul 10 16:25:26 2008 Subject: [Zope-dev] PAULA: bringing Zope 3's authentication to Plone and beyond Message-ID: <20080710202217.GD7073@marvin> Hi *, within the scope of google summer of code I am integrating zope 3's PAU with Plone's PAS and further enable (non-AT) content objects as source for users and groups. All functionality is developed in pure zope3, the plone integration is happening in a separate packages. All documents describing the project, as well as links to the code can be found here: https://chaoflow.net/projects/gsoc2008/z3membrane-ldap The code is not very interesting right now, but now would be the time to take any influence on what will be created during the next month - I am planning to continue to work on the project after the SoC. I will keep you updated on major advancements of the code. regards florian -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080710/8b8f28a8/attachment-0001.bin From wichert at wiggy.net Thu Jul 10 16:56:19 2008 From: wichert at wiggy.net (Wichert Akkerman) Date: Thu Jul 10 16:56:07 2008 Subject: [Zope-dev] Re: [Plone-developers] PAULA: bringing Zope 3's authentication to Plone and beyond In-Reply-To: <20080710202217.GD7073@marvin> References: <20080710202217.GD7073@marvin> Message-ID: <20080710205619.GB17197@wiggy.net> Previously Florian Friesdorf wrote: > Hi *, > > within the scope of google summer of code I am integrating zope 3's PAU with > Plone's PAS and further enable (non-AT) content objects as source for users and > groups. All functionality is developed in pure zope3, the plone integration is > happening in a separate packages. > > All documents describing the project, as well as links to the code can be found > here: > > https://chaoflow.net/projects/gsoc2008/z3membrane-ldap The one thing I am missing is: why? PAS works fine and covers a lot more functionality than PAU and there are more PAS plugins than PAU plugins. It's also perfectly possible to use non-AT content as source for users with PAS as well as tools such as b-org demonstrate. Wichert. -- Wichert Akkerman It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. From wichert at wiggy.net Thu Jul 10 17:07:35 2008 From: wichert at wiggy.net (Wichert Akkerman) Date: Thu Jul 10 17:07:23 2008 Subject: [Zope-dev] Re: [Plone-developers] PAULA: bringing Zope 3's authentication to Plone and beyond In-Reply-To: References: <20080710202217.GD7073@marvin> <20080710205619.GB17197@wiggy.net> Message-ID: <20080710210734.GC17197@wiggy.net> Previously Philipp von Weitershausen wrote: > Wichert Akkerman wrote: > > Previously Florian Friesdorf wrote: > >> Hi *, > >> > >> within the scope of google summer of code I am integrating zope 3's PAU with > >> Plone's PAS and further enable (non-AT) content objects as source for users and > >> groups. All functionality is developed in pure zope3, the plone integration is > >> happening in a separate packages. > >> > >> All documents describing the project, as well as links to the code can be found > >> here: > >> > >> https://chaoflow.net/projects/gsoc2008/z3membrane-ldap > > > > The one thing I am missing is: why? PAS works fine and covers a lot more > > functionality than PAU and there are more PAS plugins than PAU plugins. > > It's also perfectly possible to use non-AT content as source for users > > with PAS as well as tools such as b-org demonstrate. > > Exactly. I don't mean to pee on anybody's parade here, but IMHO Wichert > is right. To be constructive, I think it'd be much more interesting to > investigate hooking Plone up to an external authentication mechanism > such as repoze.who. Doesn't repoze already have a PAS plugin to do just that? Wichert. -- Wichert Akkerman It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. From philipp at weitershausen.de Thu Jul 10 17:01:49 2008 From: philipp at weitershausen.de (Philipp von Weitershausen) Date: Thu Jul 10 17:09:51 2008 Subject: [Zope-dev] Re: [Plone-developers] PAULA: bringing Zope 3's authentication to Plone and beyond In-Reply-To: <20080710205619.GB17197@wiggy.net> References: <20080710202217.GD7073@marvin> <20080710205619.GB17197@wiggy.net> Message-ID: Wichert Akkerman wrote: > Previously Florian Friesdorf wrote: >> Hi *, >> >> within the scope of google summer of code I am integrating zope 3's PAU with >> Plone's PAS and further enable (non-AT) content objects as source for users and >> groups. All functionality is developed in pure zope3, the plone integration is >> happening in a separate packages. >> >> All documents describing the project, as well as links to the code can be found >> here: >> >> https://chaoflow.net/projects/gsoc2008/z3membrane-ldap > > The one thing I am missing is: why? PAS works fine and covers a lot more > functionality than PAU and there are more PAS plugins than PAU plugins. > It's also perfectly possible to use non-AT content as source for users > with PAS as well as tools such as b-org demonstrate. Exactly. I don't mean to pee on anybody's parade here, but IMHO Wichert is right. To be constructive, I think it'd be much more interesting to investigate hooking Plone up to an external authentication mechanism such as repoze.who. From philipp at weitershausen.de Thu Jul 10 17:11:40 2008 From: philipp at weitershausen.de (Philipp von Weitershausen) Date: Thu Jul 10 17:14:49 2008 Subject: [Zope-dev] Re: PAULA: bringing Zope 3's authentication to Plone and beyond In-Reply-To: <20080710210734.GC17197@wiggy.net> References: <20080710202217.GD7073@marvin> <20080710205619.GB17197@wiggy.net> <20080710210734.GC17197@wiggy.net> Message-ID: Wichert Akkerman wrote: > Previously Philipp von Weitershausen wrote: >> Wichert Akkerman wrote: >>> Previously Florian Friesdorf wrote: >>>> Hi *, >>>> >>>> within the scope of google summer of code I am integrating zope 3's PAU with >>>> Plone's PAS and further enable (non-AT) content objects as source for users and >>>> groups. All functionality is developed in pure zope3, the plone integration is >>>> happening in a separate packages. >>>> >>>> All documents describing the project, as well as links to the code can be found >>>> here: >>>> >>>> https://chaoflow.net/projects/gsoc2008/z3membrane-ldap >>> The one thing I am missing is: why? PAS works fine and covers a lot more >>> functionality than PAU and there are more PAS plugins than PAU plugins. >>> It's also perfectly possible to use non-AT content as source for users >>> with PAS as well as tools such as b-org demonstrate. >> Exactly. I don't mean to pee on anybody's parade here, but IMHO Wichert >> is right. To be constructive, I think it'd be much more interesting to >> investigate hooking Plone up to an external authentication mechanism >> such as repoze.who. > > Doesn't repoze already have a PAS plugin to do just that? Ah, that's right: http://svn.agendaless.com/Products.whoopass. That doesn't mean it could be integrated better, though, for instance UI-wise... From srichter at cosmos.phy.tufts.edu Thu Jul 10 19:15:08 2008 From: srichter at cosmos.phy.tufts.edu (Stephan Richter) Date: Thu Jul 10 19:15:51 2008 Subject: [Zope-dev] RestrictedPython implementation in zope2. In-Reply-To: <20aa8c370807101045w608cdafepf9c87fca0659e3ae@mail.gmail.com> References: <20aa8c370807101045w608cdafepf9c87fca0659e3ae@mail.gmail.com> Message-ID: <200807101615.09150.srichter@cosmos.phy.tufts.edu> On Thursday 10 July 2008, ranjith kannikara wrote: > During the porting of zope2 to python2.5 I am in need and guidance on > doing the security auditing of RestrictedPython for python2.5 . Now a > person named Chris Withers had volunteered for helping. And I will be > happy to get guidance and help from Chris Withers. Since I am heavily using Python 2.5