From peter at fry-it.com  Thu May  1 08:12:55 2008
From: peter at fry-it.com (Peter Bengtsson)
Date: Thu May  1 08:12:58 2008
Subject: [Zope] Re: But I have python2.4-dev installed
In-Reply-To: <481654A7.7020204@palladion.com>
References: <bc95c2800804281533q3448b3e9p388b1c80397cc9e2@mail.gmail.com>
	<481654A7.7020204@palladion.com>
Message-ID: <bc95c2800805010512m6958cc18w2e4ac02eb50fc235@mail.gmail.com>

Turns out that I was missing libc6-dev (I had libc6)

2008/4/28 Tres Seaver <tseaver@palladion.com>:
>
>
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>
>  Peter Bengtsson wrote:
>  > Searching the mailing list it seems that the uniform advice is to
>  > install python2.4-dev but I already have that installed. I also
>  > installed the metapackage
>  > python-all-dev just to be safe. I'll paste the whole error of what
>  > happens when I try to make a configured Zope 2.10 or Zope 2.9.8.
>  > In the error it says this:
>  > "/usr/lib/gcc/i486-linux-gnu/4.2.3/include/limits.h:122:61: error:
>  > limits.h: No such file or directory"
>  > But that file exists and around line 122 it contains the following:
>  > 121> #ifdef _GCC_NEXT_LIMITS_H
>  > 122> #include_next <limits.h>                /* recurse down to the real one */
>  > 123> #endif
>  >
>  > This is the first Zope I've tried to install on this brand spanking
>  > new (K)ubuntu 8.04 which is a fresh install (not an upgrade).
>  > The error:
>
>  Googling that error reveals the following gcc thread which seems to be
>  relevant:
>
>   http://gcc.gnu.org/ml/gcc-patches/2004-11/msg00819.html
>
>
>  Tres.
>  - --
>  ===================================================================
>  Tres Seaver          +1 540-429-0999          tseaver@palladion.com
>  Palladion Software   "Excellence by Design"    http://palladion.com
>  -----BEGIN PGP SIGNATURE-----
>  Version: GnuPG v1.4.6 (GNU/Linux)
>  Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
>  iD8DBQFIFlSn+gerLs4ltQ4RArWtAJ0ceikWr8z9Ty7wnKP8rR1Sr9oGowCgy3sq
>  2/qcsS+1cQuvQ/XUNqPUQok=
>  =PYkF
>  -----END PGP SIGNATURE-----
>
>



-- 
Peter Bengtsson,
work www.fry-it.com
home www.peterbe.com
hobby www.issuetrackerproduct.com
From fbe at abes.fr  Fri May  2 05:46:43 2008
From: fbe at abes.fr (Frederique Blondelle)
Date: Fri May  2 05:46:47 2008
Subject: [Zope] problem attach file / Zfile
Message-ID: <481AE303.8040903@abes.fr>

Hello,

I have created a zclass with zfile as basic class.
With this class, users could, via a web form, send an attached file if 
they want.
With zope 2.5 (my previous version) if users don't attach a file, there 
is no problem
with zope 2.10, if there is no file attached, I have an error message : 
'file not specified". It seems that  attach  file is mandatory.

Who could help me

THanks
From fbe at abes.fr  Fri May  2 06:16:32 2008
From: fbe at abes.fr (Frederique Blondelle)
Date: Fri May  2 06:16:35 2008
Subject: [Zope] problem attach file / Zfile - more informations
Message-ID: <481AEA00.4070107@abes.fr>

Hello,

I have created a zclass with zfile as basic class.
With this class, users could, via a web form, send an attached file if
they want.
With zope 2.5 (my previous version) if users don't attach a file, there
is no problem
with zope 2.10, if there is no file attached, I have an error message :
'file not specified". It seems that  attach  file is mandatory.

Here, the complete message from the events log :

Traceback (innermost last):
   Module ZPublisher.Publish, line 119, in publish
   Module ZPublisher.mapply, line 88, in mapply
   Module ZPublisher.Publish, line 42, in call_object
   Module OFS.DTMLMethod, line 144, in __call__
    - <DTMLMethod at /assistance/do_add>
    - URL: http://assistance.abes.fr/do_add/manage_main
    - Physical Path: /assistance/do_add
   Module DocumentTemplate.DT_String, line 476, in __call__
   Module DocumentTemplate.DT_With, line 76, in render
   Module DocumentTemplate.DT_Util, line 196, in eval
    - __traceback_info__: _
   Module <string>, line 1, in <expression>
   Module OFS.DTMLMethod, line 137, in __call__
    - <DTMLMethod at /assistance/Quest_susvp_add>
    - URL: http://assistance.abes.fr/Quest_susvp_add/manage_main
    - Physical Path: /assistance/Quest_susvp_add
   Module DocumentTemplate.DT_String, line 476, in __call__
   Module DocumentTemplate.DT_With, line 61, in render
   Module DocumentTemplate.DT_Util, line 196, in eval
    - __traceback_info__: REQUEST
   Module <string>, line 1, in <expression>
   Module ZClasses.ZClass, line 454, in createInObjectManager
   Module ZClasses.ZClass, line 485, in fromRequest
   Module ZPublisher.mapply, line 89, in mapply
   Module OFS.Image, line 120, in __init__
   Module OFS.Image, line 501, in _read_data
ValueError: File not specified



Who could help me

THanks

From dev101 at magma.ca  Fri May  2 06:37:22 2008
From: dev101 at magma.ca (Jonathan)
Date: Fri May  2 06:37:23 2008
Subject: [Zope] problem attach file / Zfile - more informations
References: <481AEA00.4070107@abes.fr>
Message-ID: <000601c8ac40$8299c8a0$677ba8c0@DESKTOP>


----- Original Message ----- 
From: "Frederique Blondelle" <fbe@abes.fr>
To: <zope@zope.org>
Sent: Friday, May 02, 2008 6:16 AM
Subject: [Zope] problem attach file / Zfile - more informations


> Hello,
>
> I have created a zclass with zfile as basic class.
> With this class, users could, via a web form, send an attached file if
> they want.
> With zope 2.5 (my previous version) if users don't attach a file, there
> is no problem
> with zope 2.10, if there is no file attached, I have an error message :
> 'file not specified". It seems that  attach  file is mandatory.
>
> Here, the complete message from the events log :
>
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 119, in publish
>   Module ZPublisher.mapply, line 88, in mapply
>   Module ZPublisher.Publish, line 42, in call_object
>   Module OFS.DTMLMethod, line 144, in __call__
>    - <DTMLMethod at /assistance/do_add>
>    - URL: http://assistance.abes.fr/do_add/manage_main
>    - Physical Path: /assistance/do_add
>   Module DocumentTemplate.DT_String, line 476, in __call__
>   Module DocumentTemplate.DT_With, line 76, in render
>   Module DocumentTemplate.DT_Util, line 196, in eval
>    - __traceback_info__: _
>   Module <string>, line 1, in <expression>
>   Module OFS.DTMLMethod, line 137, in __call__
>    - <DTMLMethod at /assistance/Quest_susvp_add>
>    - URL: http://assistance.abes.fr/Quest_susvp_add/manage_main
>    - Physical Path: /assistance/Quest_susvp_add
>   Module DocumentTemplate.DT_String, line 476, in __call__
>   Module DocumentTemplate.DT_With, line 61, in render
>   Module DocumentTemplate.DT_Util, line 196, in eval
>    - __traceback_info__: REQUEST
>   Module <string>, line 1, in <expression>
>   Module ZClasses.ZClass, line 454, in createInObjectManager
>   Module ZClasses.ZClass, line 485, in fromRequest
>   Module ZPublisher.mapply, line 89, in mapply
>   Module OFS.Image, line 120, in __init__
>   Module OFS.Image, line 501, in _read_data
> ValueError: File not specified

I don't know if this is the cause of your problem, but as of Zope 2.10 
ZClasses are deprecated.


Jonathan 

From mva.led at gmail.com  Sun May  4 09:44:17 2008
From: mva.led at gmail.com (Manuel Vazquez Acosta)
Date: Sun May  4 09:44:28 2008
Subject: [Zope] Zope threads and the GIL
Message-ID: <481DBDB1.5050606@gmail.com>

Hi all, How does the GIL affect Zope threads?

Best regards,
Manuel.
From lists at zopyx.com  Sun May  4 09:51:34 2008
From: lists at zopyx.com (Andreas Jung)
Date: Sun May  4 09:51:42 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <481DBDB1.5050606@gmail.com>
References: <481DBDB1.5050606@gmail.com>
Message-ID: <3FB1BA0FFA6AC60C4A01D94F@[192.168.0.24]>



--On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta <mva.led@gmail.com> 
wrote:

> Hi all, How does the GIL affect Zope threads?

Zope threads are Python threads and therefore a thread switch is not 
possible is some thread is holding the GIL.

-aj

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080504/ac3ede8b/attachment.bin
From mva.led at gmail.com  Sun May  4 11:12:16 2008
From: mva.led at gmail.com (Manuel Vazquez Acosta)
Date: Sun May  4 11:12:29 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <3FB1BA0FFA6AC60C4A01D94F@[192.168.0.24]>
References: <481DBDB1.5050606@gmail.com>
	<3FB1BA0FFA6AC60C4A01D94F@[192.168.0.24]>
Message-ID: <481DD250.2040509@gmail.com>

Andreas Jung wrote:
> 
> 
> --On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta
> <mva.led@gmail.com> wrote:
> 
>> Hi all, How does the GIL affect Zope threads?
> 
> Zope threads are Python threads and therefore a thread switch is not
> possible is some thread is holding the GIL.
> 
> -aj
> 

That would mean that the only place where real multitasking occurs is
when fetching objects from ZEO and other I/O bounded tasks, isn't?

Best regards,
Manuel
From python at zeomega.com  Sun May  4 20:52:38 2008
From: python at zeomega.com (sam)
Date: Sun May  4 20:53:48 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <481DD250.2040509@gmail.com>
Message-ID: <357262106.76521209948758058.JavaMail.root@chandra.zeomega.com>

another scenario is  if you are using GIL aware Database adapters like mxODBC to access  an RDBMS 

sam
----- Original Message -----
From: "Manuel Vazquez Acosta" <mva.led@gmail.com>
To: "Andreas Jung" <lists@zopyx.com>
Cc: "Zope user list" <zope@zope.org>
Sent: Sunday, May 4, 2008 9:12:16 AM (GMT-0600) America/Guatemala
Subject: Re: [Zope] Zope threads and the GIL

Andreas Jung wrote:
> 
> 
> --On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta
> <mva.led@gmail.com> wrote:
> 
>> Hi all, How does the GIL affect Zope threads?
> 
> Zope threads are Python threads and therefore a thread switch is not
> possible is some thread is holding the GIL.
> 
> -aj
> 

That would mean that the only place where real multitasking occurs is
when fetching objects from ZEO and other I/O bounded tasks, isn't?

Best regards,
Manuel
_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

From lists at zopyx.com  Sun May  4 21:36:08 2008
From: lists at zopyx.com (Andreas Jung)
Date: Sun May  4 21:36:19 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <357262106.76521209948758058.JavaMail.root@chandra.zeomega.com>
References: <357262106.76521209948758058.JavaMail.root@chandra.zeomega.com>
Message-ID: <71DF21E409D29A6A19DFEE41@[192.168.0.24]>



--On 4. Mai 2008 19:52:38 -0500 sam <python@zeomega.com> wrote:

> another scenario is  if you are using GIL aware Database adapters like
> mxODBC to access  an RDBMS
>

What means GIL-aware in this context?

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080505/fba733d5/attachment.bin
From python at zeomega.com  Sun May  4 22:05:55 2008
From: python at zeomega.com (sam)
Date: Sun May  4 22:07:06 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <71DF21E409D29A6A19DFEE41@[192.168.0.24]>
Message-ID: <1491934427.76641209953155068.JavaMail.root@chandra.zeomega.com>

Basically mxODBC is a third party extension and is GIL "aware" as it yields the GIL before accessing the DB

----- Original Message -----
From: "Andreas Jung" <lists@zopyx.com>
To: "sam" <python@zeomega.com>, "Manuel Vazquez Acosta" <mva.led@gmail.com>
Cc: "Zope user list" <zope@zope.org>
Sent: Sunday, May 4, 2008 7:36:08 PM (GMT-0600) America/Guatemala
Subject: Re: [Zope] Zope threads and the GIL



--On 4. Mai 2008 19:52:38 -0500 sam <python@zeomega.com> wrote:

> another scenario is  if you are using GIL aware Database adapters like
> mxODBC to access  an RDBMS
>

What means GIL-aware in this context?

-aj
From lists at zopyx.com  Sun May  4 22:34:49 2008
From: lists at zopyx.com (Andreas Jung)
Date: Sun May  4 22:34:56 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <1491934427.76641209953155068.JavaMail.root@chandra.zeomega.com>
References: <1491934427.76641209953155068.JavaMail.root@chandra.zeomega.com>
Message-ID: <E7D097414FA4CC8682AF948E@[192.168.0.24]>



--On 4. Mai 2008 21:05:55 -0500 sam <python@zeomega.com> wrote:

> Basically mxODBC is a third party extension and is GIL "aware" as it
> yields the GIL before accessing the DB

Then it should play nicely with Python. But keep in mind that thread 
switches can also be blocked by some thread without having the GIL locked.
We've seen such situations were a thread had to perform complex regular 
expression matching operations. Obviously code in Python C extension is 
also able to lock up the Python interpreter until the result is being 
passed back to the Python interpreter.

Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080505/85f8195e/attachment.bin
From mva.led at gmail.com  Mon May  5 10:11:32 2008
From: mva.led at gmail.com (Manuel Vazquez Acosta)
Date: Mon May  5 10:11:45 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <E7D097414FA4CC8682AF948E@[192.168.0.24]>
References: <1491934427.76641209953155068.JavaMail.root@chandra.zeomega.com>
	<E7D097414FA4CC8682AF948E@[192.168.0.24]>
Message-ID: <481F1594.2060807@gmail.com>

Andreas Jung wrote:
> Then it should play nicely with Python. But keep in mind that thread
> switches can also be blocked by some thread without having the GIL locked.
> We've seen such situations were a thread had to perform complex regular
> expression matching operations. Obviously code in Python C extension is
> also able to lock up the Python interpreter until the result is being
> passed back to the Python interpreter.
> 

Another Q: Is AccessControl GIL aware? I mean, this is probably the most
used component in Zope. Maybe a performance boost is experienced if
those checks can be made simultaneously.

Thinking harder, I realize that a tradeoff is in place: Objects can
change its security status (the security policy -- i.e, the security
object -- can be changed also, but is a weird use case, isn't it?). So
maybe write-lock is needed.

On the other hand, for objects with a high demand rate, which are kept
in memory, good results are to be expected.

CacheFu deals OK with Anonymous users, but a boost is to be expected for
authenticated users.

Best regards,
Manuel.
From lists at zopyx.com  Mon May  5 11:26:42 2008
From: lists at zopyx.com (Andreas Jung)
Date: Mon May  5 11:27:03 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <481F1594.2060807@gmail.com>
References: <1491934427.76641209953155068.JavaMail.root@chandra.zeomega.com>
	<E7D097414FA4CC8682AF948E@[192.168.0.24]> <481F1594.2060807@gmail.com>
Message-ID: <44F3278D31CEE8DAC2ACA08E@suxmac.local>



--On 5. Mai 2008 10:11:32 -0400 Manuel Vazquez Acosta <mva.led@gmail.com> 
wrote:

> Andreas Jung wrote:
>> Then it should play nicely with Python. But keep in mind that thread
>> switches can also be blocked by some thread without having the GIL
>> locked. We've seen such situations were a thread had to perform complex
>> regular expression matching operations. Obviously code in Python C
>> extension is also able to lock up the Python interpreter until the
>> result is being passed back to the Python interpreter.
>>
>
> Another Q: Is AccessControl GIL aware? I mean, this is probably the most
> used component in Zope. Maybe a performance boost is experienced if
> those checks can be made simultaneously.

Better tell us what your _real_ question/problem is. I am not getting the 
point what you are really interested in.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080505/97b8de81/attachment.bin
From mva.led at gmail.com  Mon May  5 11:47:33 2008
From: mva.led at gmail.com (Manuel Vazquez Acosta)
Date: Mon May  5 11:47:47 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <44F3278D31CEE8DAC2ACA08E@suxmac.local>
References: <1491934427.76641209953155068.JavaMail.root@chandra.zeomega.com>
	<E7D097414FA4CC8682AF948E@[192.168.0.24]>
	<481F1594.2060807@gmail.com>
	<44F3278D31CEE8DAC2ACA08E@suxmac.local>
Message-ID: <481F2C15.2060200@gmail.com>

Andreas Jung wrote:
> 
> 
> --On 5. Mai 2008 10:11:32 -0400 Manuel Vazquez Acosta
> <mva.led@gmail.com> wrote:
> 
>> Andreas Jung wrote:
>>> Then it should play nicely with Python. But keep in mind that thread
>>> switches can also be blocked by some thread without having the GIL
>>> locked. We've seen such situations were a thread had to perform complex
>>> regular expression matching operations. Obviously code in Python C
>>> extension is also able to lock up the Python interpreter until the
>>> result is being passed back to the Python interpreter.
>>>
>>
>> Another Q: Is AccessControl GIL aware? I mean, this is probably the most
>> used component in Zope. Maybe a performance boost is experienced if
>> those checks can be made simultaneously.
> 
> Better tell us what your _real_ question/problem is. I am not getting
> the point what you are really interested in.
> 
> -aj

I'm thinking in CPU-bounded parts of a request: Reindexing, Clustering,
Automatic Content Extraction, and the like...

We have implemented k-means clustering algorithm on top ZCTextIndex (now
plan to port to zc.catalog), although this is batch operation, some
other are per-query: Summarizing is one of those: Per cluster, and per
document. (Although this is maybe a question for the OTS list).

So I'm a little bit worried about how the resource utilization.

Best regards,
Manuel.




From dieter at handshake.de  Mon May  5 15:29:40 2008
From: dieter at handshake.de (Dieter Maurer)
Date: Mon May  5 15:30:01 2008
Subject: [Zope] Add Combo Box in External method page
In-Reply-To: <18a05b120804292256p3c93bea7ye56e82d662f64104@mail.gmail.com>
References: <18a05b120804292256p3c93bea7ye56e82d662f64104@mail.gmail.com>
Message-ID: <18463.24612.832906.327653@gargle.gargle.HOWL>

vaibhav pol wrote at 2008-4-30 11:26 +0530:
> ...
> I want to modify the ExternalMethod dtml page which cames in zope i have to
>add combobox into the external method
> i go into the
> zopehome/lib/python/Products/ExternalMethod/dtml where i get the dtml page
>methodAdd.dtml
> i add combo box this dtml page is call in ExternalMethod.py how can i
>modifed that is it i have to create a object for the combobox .

I fear I cannot understand this English.

Where is your problem?

In "methodAdd.dtml" when you try to construct the combobox?
In the corresponding action when you try to interpret and handle
the value from your combobox?



-- 
Dieter
From dieter at handshake.de  Mon May  5 15:34:28 2008
From: dieter at handshake.de (Dieter Maurer)
Date: Mon May  5 15:34:50 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <481DD250.2040509@gmail.com>
References: <481DBDB1.5050606@gmail.com>
	<3FB1BA0FFA6AC60C4A01D94F@[192.168.0.24]>
	<481DD250.2040509@gmail.com>
Message-ID: <18463.24900.872601.24796@gargle.gargle.HOWL>

Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
> ...
>That would mean that the only place where real multitasking occurs is
>when fetching objects from ZEO and other I/O bounded tasks, isn't?

I depends what you mean with "real multitasking".

If you mean by this that several CPUs are concurrently active for your
Zope process, then you are near. You can, however, also write code
for expensive time consuming operations in a language different from
Python and then release the GIL before you active such code. Then,
this code, too, can occupy a CPU in addition to the Python activity.



-- 
Dieter
From mva.led at gmail.com  Mon May  5 18:57:54 2008
From: mva.led at gmail.com (Manuel Vazquez Acosta)
Date: Mon May  5 18:58:08 2008
Subject: [Zope] Zope threads and the GIL
In-Reply-To: <18463.24900.872601.24796@gargle.gargle.HOWL>
References: <481DBDB1.5050606@gmail.com>	<3FB1BA0FFA6AC60C4A01D94F@[192.168.0.24]>	<481DD250.2040509@gmail.com>
	<18463.24900.872601.24796@gargle.gargle.HOWL>
Message-ID: <481F90F2.40202@gmail.com>

Dieter Maurer wrote:
> Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
>> ...
>> That would mean that the only place where real multitasking occurs is
>> when fetching objects from ZEO and other I/O bounded tasks, isn't?
> 
> I depends what you mean with "real multitasking".
> 
> If you mean by this that several CPUs are concurrently active for your
> Zope process, then you are near. 

Exactly. Several CPUs, I would like to use.

> You can, however, also write code
> for expensive time consuming operations in a language different from
> Python and then release the GIL before you active such code. Then,
> this code, too, can occupy a CPU in addition to the Python activity.
> 

I'm very new to Python, and don't grasp yet the C API. Also I would like
to implement almost everything in Python: I need this, b/c my of my
team's skills in C are None.

Best regards,
Manuel.

From tseaver at palladion.com  Mon May  5 22:02:02 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Mon May  5 22:02:18 2008
Subject: [Zope] Re: Zope threads and the GIL
In-Reply-To: <481F90F2.40202@gmail.com>
References: <481DBDB1.5050606@gmail.com>	<3FB1BA0FFA6AC60C4A01D94F@[192.168.0.24]>	<481DD250.2040509@gmail.com>	<18463.24900.872601.24796@gargle.gargle.HOWL>
	<481F90F2.40202@gmail.com>
Message-ID: <481FBC1A.9070105@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Manuel Vazquez Acosta wrote:
> Dieter Maurer wrote:
>> Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
>>> ...
>>> That would mean that the only place where real multitasking occurs is
>>> when fetching objects from ZEO and other I/O bounded tasks, isn't?
>> I depends what you mean with "real multitasking".
>>
>> If you mean by this that several CPUs are concurrently active for your
>> Zope process, then you are near. 
> 
> Exactly. Several CPUs, I would like to use.

Generally, the rule-of-thumb for scaling Zope on a multi-CPU box is to
run an appserver instance per CPU, each talking to a shared storage
(ZEO, usually, but RelStorage would do as well).


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver@palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIH7wa+gerLs4ltQ4RAndLAJwM+qs7w5hQy0tA9ZY33mJBGPeTBgCeN8fi
J50GVau5mZLxpFnRyJLYCIc=
=09EA
-----END PGP SIGNATURE-----

From D.Sinang at spi-bpo.com  Mon May  5 22:28:29 2008
From: D.Sinang at spi-bpo.com (Sinang, Danny)
Date: Mon May  5 22:28:39 2008
Subject: [Zope] Turning back system clock
Message-ID: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C0E@spi-mail2003.spitech.com>

Dear All,
 
The system clock of our Zope 2.7 machine accidentally got set to 2009.
We noticed this mistake 6 hours later and we decided to set back the
clock to 2008. 
 
How does this affect the integrity of Data.fs ?
 
>From our observation :
 
1. Session write errors suddenly began to appear
 
2. Data.fs.index won't grow anymore and its timestamp remained at the
time we set back the clock
 
3. Packing the database (which was 8 GB at the time) took too long
without any signs of any Data.fs.pack file in the var folder. We decided
to restart Zope after a few minutes.
 
4. There seem to be some catalog indexing problems as our catalog
searches on a certain field doesn't work as reliably as it did before.
We tried reindexing the field and even the catalog, but it timed out on
us (lots of objects in the database).
 
Has anyone here had any similar experiences ? How do I go about fixing
this ?
 
Using repozo.py last night, I made a full backup of Data.fs, copied it
to a new instance, was able to create a new Data.fs.index, and
eventually succeeded in packing the 8 GB database to just 2.4 GB. But it
took quite a while - like 10 or 15 minutes before any Data.fs.pack
appeared. 
 
I haven't checked yet if the catalog indexing problem has gone away.
 
Regards,
Danny
 
 
 
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20080506/0ec5e880/attachment.html
From D.Sinang at spi-bpo.com  Tue May  6 00:19:10 2008
From: D.Sinang at spi-bpo.com (Sinang, Danny)
Date: Tue May  6 00:19:21 2008
Subject: [Zope] How to stop packing ?
Message-ID: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C56@spi-mail2003.spitech.com>

Hi All,
 
Is it possible to stop an ongoing ZODB packing operation without
restarting Zope ?
 
Am using Zope 2.7.6-final.
 
- Danny
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20080506/40e99cef/attachment.html
From lists at zopyx.com  Tue May  6 00:29:17 2008
From: lists at zopyx.com (Andreas Jung)
Date: Tue May  6 00:29:29 2008
Subject: [Zope] How to stop packing ?
In-Reply-To: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C56@spi-mail2003.spitech.com>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C56@spi-mail2003.spitech.com>
Message-ID: <48D059233E3D17F8F2802916@suxmac.local>



--On 6. Mai 2008 12:19:10 +0800 "Sinang, Danny" <D.Sinang@spi-bpo.com> 
wrote:

> Hi All,
>
> Is it possible to stop an ongoing ZODB packing operation without
> restarting Zope ?
>
>

Restart.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080506/3d7e63b1/attachment.bin
From lists at zopyx.com  Tue May  6 00:31:23 2008
From: lists at zopyx.com (Andreas Jung)
Date: Tue May  6 00:31:34 2008
Subject: [Zope] Turning back system clock
In-Reply-To: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C0E@spi-mail2003.spitech.com>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C0E@spi-mail2003.spitech.com>
Message-ID: <F03AEC2CD13342088B37D402@suxmac.local>



--On 6. Mai 2008 10:28:29 +0800 "Sinang, Danny" <D.Sinang@spi-bpo.com> 
wrote:

> Dear All,
>
> The system clock of our Zope 2.7 machine accidentally got set to 2009.
> We noticed this mistake 6 hours later and we decided to set back the
> clock to 2008.
>
> How does this affect the integrity of Data.fs ?
>
>> From our observation :
>
> 1. Session write errors suddenly began to appear
>
> 2. Data.fs.index won't grow anymore and its timestamp remained at the
> time we set back the clock
>
> 3. Packing the database (which was 8 GB at the time) took too long
> without any signs of any Data.fs.pack file in the var folder. We decided
> to restart Zope after a few minutes.
>
> 4. There seem to be some catalog indexing problems as our catalog
> searches on a certain field doesn't work as reliably as it did before.
> We tried reindexing the field and even the catalog, but it timed out on
> us (lots of objects in the database).
>
> Has anyone here had any similar experiences ? How do I go about fixing
> this ?

Turning the clock back is known to result in data inconsistencies.
It is is best-practice having NTP installed on all Zope machines.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080506/91f25785/attachment.bin
From D.Sinang at spi-bpo.com  Tue May  6 00:37:50 2008
From: D.Sinang at spi-bpo.com (Sinang, Danny)
Date: Tue May  6 00:38:04 2008
Subject: [Zope] Turning back system clock
In-Reply-To: <F03AEC2CD13342088B37D402@suxmac.local>
Message-ID: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C65@spi-mail2003.spitech.com>

Thanks.

Is there a way to spot these data inconsistencies in Data.fs and remove
or fix them ?

Regards,
Danny 

-----Original Message-----
From: Andreas Jung [mailto:lists@zopyx.com] 
Sent: Tuesday, May 06, 2008 12:31 PM
To: Sinang, Danny; zope@zope.org
Subject: Re: [Zope] Turning back system clock



--On 6. Mai 2008 10:28:29 +0800 "Sinang, Danny" <D.Sinang@spi-bpo.com>
wrote:

> Dear All,
>
> The system clock of our Zope 2.7 machine accidentally got set to 2009.
> We noticed this mistake 6 hours later and we decided to set back the 
> clock to 2008.
>
> How does this affect the integrity of Data.fs ?
>
>> From our observation :
>
> 1. Session write errors suddenly began to appear
>
> 2. Data.fs.index won't grow anymore and its timestamp remained at the 
> time we set back the clock
>
> 3. Packing the database (which was 8 GB at the time) took too long 
> without any signs of any Data.fs.pack file in the var folder. We 
> decided to restart Zope after a few minutes.
>
> 4. There seem to be some catalog indexing problems as our catalog 
> searches on a certain field doesn't work as reliably as it did before.
> We tried reindexing the field and even the catalog, but it timed out 
> on us (lots of objects in the database).
>
> Has anyone here had any similar experiences ? How do I go about fixing

> this ?

Turning the clock back is known to result in data inconsistencies.
It is is best-practice having NTP installed on all Zope machines.

-aj
From lists at zopyx.com  Tue May  6 00:57:57 2008
From: lists at zopyx.com (Andreas Jung)
Date: Tue May  6 00:58:08 2008
Subject: [Zope] Turning back system clock
In-Reply-To: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C65@spi-mail2003.spitech.com>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C65@spi-mail2003.spitech.com>
Message-ID: <B807CABBA9A94F843B68CD80@suxmac.local>



--On 6. Mai 2008 12:37:50 +0800 "Sinang, Danny" <D.Sinang@spi-bpo.com> 
wrote:

> Thanks.
>
> Is there a way to spot these data inconsistencies in Data.fs and remove
> or fix them ?
>

You might ask Google. This topic came up several times in the past.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080506/fcb2a19c/attachment.bin
From garry at schoolteachers.co.uk  Wed May  7 11:52:35 2008
From: garry at schoolteachers.co.uk (Garry Saddington)
Date: Wed May  7 11:46:25 2008
Subject: [Zope] disappearing external methods
Message-ID: <200805071652.35998.garry@schoolteachers.co.uk>

I have recently moved our Zope installation over to a centos server. Since the 
move all the external methods periodically stop working for no apparent 
reason and the error is returned as:
"external method can not be called because it is none"

The thing is that it works well and then stops. Does anyone have any ideas 
about this? The installation was not initially compiled on centos as I have a 
large collection of software as part of the application which can be moved 
between servers with the Zope installation. The reason for the move to Centos 
is that DTMLTex fails when trying to run pdflatex on Ubuntu and this seems to 
be distro specific, with no apparent way to fix it, unless anyone knows any 
better?
Kind Regards
Garry
From dev101 at magma.ca  Wed May  7 11:54:01 2008
From: dev101 at magma.ca (Jonathan)
Date: Wed May  7 11:53:58 2008
Subject: [Zope] disappearing external methods
References: <200805071652.35998.garry@schoolteachers.co.uk>
Message-ID: <029701c8b05a$919c8b80$677ba8c0@DESKTOP>


----- Original Message ----- 
From: "Garry Saddington" <garry@schoolteachers.co.uk>
To: <zope@zope.org>
Sent: Wednesday, May 07, 2008 11:52 AM
Subject: [Zope] disappearing external methods


>I have recently moved our Zope installation over to a centos server. Since 
>the
> move all the external methods periodically stop working for no apparent
> reason and the error is returned as:
> "external method can not be called because it is none"
>
> The thing is that it works well and then stops. Does anyone have any ideas
> about this? The installation was not initially compiled on centos as I 
> have a
> large collection of software as part of the application which can be moved
> between servers with the Zope installation. The reason for the move to 
> Centos
> is that DTMLTex fails when trying to run pdflatex on Ubuntu and this seems 
> to
> be distro specific, with no apparent way to fix it, unless anyone knows 
> any
> better?

We use centOS and have a lot of external methods (all working happily).

Have you tried deleting the .pyc files associated with the external methods 
and then restarting zope?


Jonathan 

From dev101 at magma.ca  Wed May  7 11:56:45 2008
From: dev101 at magma.ca (Jonathan)
Date: Wed May  7 11:56:44 2008
Subject: [Zope] disappearing external methods
References: <200805071652.35998.garry@schoolteachers.co.uk>
	<029701c8b05a$919c8b80$677ba8c0@DESKTOP>
Message-ID: <029b01c8b05a$f3be9d80$677ba8c0@DESKTOP>


----- Original Message ----- 
From: "Jonathan" <dev101@magma.ca>
To: "Garry Saddington" <garry@schoolteachers.co.uk>; <zope@zope.org>
Sent: Wednesday, May 07, 2008 11:54 AM
Subject: Re: [Zope] disappearing external methods


>
> ----- Original Message ----- 
> From: "Garry Saddington" <garry@schoolteachers.co.uk>
> To: <zope@zope.org>
> Sent: Wednesday, May 07, 2008 11:52 AM
> Subject: [Zope] disappearing external methods
>
>
>>I have recently moved our Zope installation over to a centos server. Since 
>>the
>> move all the external methods periodically stop working for no apparent
>> reason and the error is returned as:
>> "external method can not be called because it is none"
>>
>> The thing is that it works well and then stops. Does anyone have any 
>> ideas
>> about this? The installation was not initially compiled on centos as I 
>> have a
>> large collection of software as part of the application which can be 
>> moved
>> between servers with the Zope installation. The reason for the move to 
>> Centos
>> is that DTMLTex fails when trying to run pdflatex on Ubuntu and this 
>> seems to
>> be distro specific, with no apparent way to fix it, unless anyone knows 
>> any
>> better?
>
> We use centOS and have a lot of external methods (all working happily).
>
> Have you tried deleting the .pyc files associated with the external 
> methods and then restarting zope?

Sorry, that should have been delete any .pyc files of modules that your 
external methods call and delete .pyc files for any products that may be 
used by the external methods, then restart zope.

You can also try to delete and re-add the external method within the ZMI.

Jonathan 

From garry at schoolteachers.co.uk  Wed May  7 12:06:11 2008
From: garry at schoolteachers.co.uk (Garry Saddington)
Date: Wed May  7 11:59:59 2008
Subject: [Zope] disappearing external methods
In-Reply-To: <029701c8b05a$919c8b80$677ba8c0@DESKTOP>
References: <200805071652.35998.garry@schoolteachers.co.uk>
	<029701c8b05a$919c8b80$677ba8c0@DESKTOP>
Message-ID: <200805071706.11929.garry@schoolteachers.co.uk>

On Wednesday 07 May 2008 16:54, Jonathan wrote:
> ----- Original Message -----
> From: "Garry Saddington" <garry@schoolteachers.co.uk>
> To: <zope@zope.org>
> Sent: Wednesday, May 07, 2008 11:52 AM
> Subject: [Zope] disappearing external methods
>
> >I have recently moved our Zope installation over to a centos server. Since
> >the
> > move all the external methods periodically stop working for no apparent
> > reason and the error is returned as:
> > "external method can not be called because it is none"
> >
> > The thing is that it works well and then stops. Does anyone have any
> > ideas about this? The installation was not initially compiled on centos
> > as I have a
> > large collection of software as part of the application which can be
> > moved between servers with the Zope installation. The reason for the move
> > to Centos
> > is that DTMLTex fails when trying to run pdflatex on Ubuntu and this
> > seems to
> > be distro specific, with no apparent way to fix it, unless anyone knows
> > any
> > better?
>
> We use centOS and have a lot of external methods (all working happily).
>
> Have you tried deleting the .pyc files associated with the external methods
> and then restarting zope?
There are no .pyc files in any of our installations.
Have you used the Zope that comes with Centos or from Zope.org?
Regards
Garry
From dev101 at magma.ca  Wed May  7 12:08:12 2008
From: dev101 at magma.ca (Jonathan)
Date: Wed May  7 12:08:10 2008
Subject: [Zope] disappearing external methods
References: <200805071652.35998.garry@schoolteachers.co.uk><029701c8b05a$919c8b80$677ba8c0@DESKTOP>
	<200805071706.11929.garry@schoolteachers.co.uk>
Message-ID: <029f01c8b05c$8d145960$677ba8c0@DESKTOP>


----- Original Message ----- 
From: "Garry Saddington" <garry@schoolteachers.co.uk>
To: <zope@zope.org>
Sent: Wednesday, May 07, 2008 12:06 PM
Subject: Re: [Zope] disappearing external methods


> On Wednesday 07 May 2008 16:54, Jonathan wrote:
>> ----- Original Message -----
>> From: "Garry Saddington" <garry@schoolteachers.co.uk>
>> To: <zope@zope.org>
>> Sent: Wednesday, May 07, 2008 11:52 AM
>> Subject: [Zope] disappearing external methods
>>
>> >I have recently moved our Zope installation over to a centos server. 
>> >Since
>> >the
>> > move all the external methods periodically stop working for no apparent
>> > reason and the error is returned as:
>> > "external method can not be called because it is none"
>> >
>> > The thing is that it works well and then stops. Does anyone have any
>> > ideas about this? The installation was not initially compiled on centos
>> > as I have a
>> > large collection of software as part of the application which can be
>> > moved between servers with the Zope installation. The reason for the 
>> > move
>> > to Centos
>> > is that DTMLTex fails when trying to run pdflatex on Ubuntu and this
>> > seems to
>> > be distro specific, with no apparent way to fix it, unless anyone knows
>> > any
>> > better?
>>
>> We use centOS and have a lot of external methods (all working happily).
>>
>> Have you tried deleting the .pyc files associated with the external 
>> methods
>> and then restarting zope?
> There are no .pyc files in any of our installations.
> Have you used the Zope that comes with Centos or from Zope.org?
> Regards
> Garry

We downloaded/installed zope from zope.org.


Jonathan 

From jmeile at hotmail.com  Wed May  7 13:13:49 2008
From: jmeile at hotmail.com (Josef Meile)
Date: Wed May  7 13:14:06 2008
Subject: [Zope] Re: disappearing external methods
In-Reply-To: <200805071706.11929.garry@schoolteachers.co.uk>
References: <200805071652.35998.garry@schoolteachers.co.uk>	<029701c8b05a$919c8b80$677ba8c0@DESKTOP>
	<200805071706.11929.garry@schoolteachers.co.uk>
Message-ID: <4821E34D.8080106@hotmail.com>

Garry Saddington schrieb:
>> Have you tried deleting the .pyc files associated with the external methods
>> and then restarting zope?
> There are no .pyc files in any of our installations.
> Have you used the Zope that comes with Centos or from Zope.org?
Tried, as somebody say, to open the external method on the ZMI, then 
press save, and test if it works. If you have a lot of them, you could
try Dieter's advice here:

http://mail.zope.org/pipermail/zope/2002-July/119101.html

Check also that you move all .py files from your external methods and
that file and folder permissions allow the user running zope to access
them.

I'm not sure but I think this problems with the external methods only
happen with old zope versions. Which version you have?

Best regards
Josef
From garry at schoolteachers.co.uk  Wed May  7 16:05:03 2008
From: garry at schoolteachers.co.uk (Garry Saddington)
Date: Wed May  7 15:58:51 2008
Subject: [Zope] Re: disappearing external methods
In-Reply-To: <4821E34D.8080106@hotmail.com>
References: <200805071652.35998.garry@schoolteachers.co.uk>
	<200805071706.11929.garry@schoolteachers.co.uk>
	<4821E34D.8080106@hotmail.com>
Message-ID: <200805072105.03299.garry@schoolteachers.co.uk>

On Wednesday 07 May 2008 18:13, Josef Meile wrote:
> Garry Saddington schrieb:
> >> Have you tried deleting the .pyc files associated with the external
> >> methods and then restarting zope?
> >
> > There are no .pyc files in any of our installations.
> > Have you used the Zope that comes with Centos or from Zope.org?
>
> Tried, as somebody say, to open the external method on the ZMI, then
> press save, and test if it works. If you have a lot of them, you could
> try Dieter's advice here:
>
> http://mail.zope.org/pipermail/zope/2002-July/119101.html
>
> Check also that you move all .py files from your external methods and
> that file and folder permissions allow the user running zope to access
> them.
>
> I'm not sure but I think this problems with the external methods only
> happen with old zope versions. Which version you have?

2.9.6
I am re-compiling on Centos to see if that helps

Regards
Garry
From D.Sinang at spi-bpo.com  Wed May  7 20:08:28 2008
From: D.Sinang at spi-bpo.com (Sinang, Danny)
Date: Wed May  7 20:09:00 2008
Subject: [Zope] Making Zope stop storing old object versions
Message-ID: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2134@spi-mail2003.spitech.com>

Hello,
 
We've got lots of Catalog reads and writes and our Data.fs is growing at
an immense rate. 
 
Is there a way we can tell Zope to stop storing old object versions to
limit the growth and not have to pack that often ?
 
Regards,
Danny
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20080508/0c8b3547/attachment.html
From tino at wildenhain.de  Wed May  7 22:54:54 2008
From: tino at wildenhain.de (Tino Wildenhain)
Date: Wed May  7 22:54:59 2008
Subject: [Zope] Making Zope stop storing old object versions
In-Reply-To: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2134@spi-mail2003.spitech.com>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2134@spi-mail2003.spitech.com>
Message-ID: <48226B7E.2000409@wildenhain.de>

Hi,

Sinang, Danny wrote:
> Hello,
>  
> We've got lots of Catalog reads and writes and our Data.fs is growing 
> at an immense rate. 
>  
> Is there a way we can tell Zope to stop storing old object 
> versions to limit the growth and not have to pack that often ?

Well in fact it (Filestorage) does only "store" new object versions.
What it does not is removing the old version - this is done during packing.

When you have many Catalog writes or other parts of your object tree
experiences many writes, try to mount an extra ZODB (and storage) there.

You can also try some of the other storages like relstorage (where it
depends on the RDBMS how it handles writes)

Cheers
Tino
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3241 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mail.zope.org/pipermail/zope/attachments/20080508/aad84a57/smime.bin
From dev101 at magma.ca  Thu May  8 07:16:23 2008
From: dev101 at magma.ca (Jonathan)
Date: Thu May  8 07:16:22 2008
Subject: [Zope] Making Zope stop storing old object versions
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2134@spi-mail2003.spitech.com>
	<48226B7E.2000409@wildenhain.de>
Message-ID: <001c01c8b0fc$f344d200$677ba8c0@DESKTOP>

>> Sinang, Danny wrote:
>> Hello,
>>
>> We've got lots of Catalog reads and writes and our Data.fs is growing
>> at an immense rate.
>>
>> Is there a way we can tell Zope to stop storing old object
>> versions to limit the growth and not have to pack that often ?
>
>Well in fact it (Filestorage) does only "store" new object versions.
>What it does not is removing the old version - this is done during packing.
>
>When you have many Catalog writes or other parts of your object tree
>experiences many writes, try to mount an extra ZODB (and storage) there.
>
>You can also try some of the other storages like relstorage (where it
>depends on the RDBMS how it handles writes)

Alternatively, you could just replace your Catalog with an external db such 
as MySQL (using zmysql database connections).


Jonathan 

From D.Sinang at spi-bpo.com  Thu May  8 10:03:55 2008
From: D.Sinang at spi-bpo.com (Sinang, Danny)
Date: Thu May  8 10:04:02 2008
Subject: [Zope] Making Zope stop storing old object versions
In-Reply-To: <001c01c8b0fc$f344d200$677ba8c0@DESKTOP>
Message-ID: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2307@spi-mail2003.spitech.com>

How do I go about replacing a Catalog using MySQL ? This is the first
time I've heard this. 

-----Original Message-----
From: Jonathan [mailto:dev101@magma.ca] 
Sent: Thursday, May 08, 2008 7:16 PM
To: Sinang, Danny
Cc: zope@zope.org
Subject: Re: [Zope] Making Zope stop storing old object versions

>> Sinang, Danny wrote:
>> Hello,
>>
>> We've got lots of Catalog reads and writes and our Data.fs is growing

>> at an immense rate.
>>
>> Is there a way we can tell Zope to stop storing old object versions 
>> to limit the growth and not have to pack that often ?
>
>Well in fact it (Filestorage) does only "store" new object versions.
>What it does not is removing the old version - this is done during
packing.
>
>When you have many Catalog writes or other parts of your object tree 
>experiences many writes, try to mount an extra ZODB (and storage)
there.
>
>You can also try some of the other storages like relstorage (where it 
>depends on the RDBMS how it handles writes)

Alternatively, you could just replace your Catalog with an external db
such as MySQL (using zmysql database connections).


Jonathan 

From dev101 at magma.ca  Thu May  8 10:08:26 2008
From: dev101 at magma.ca (Jonathan)
Date: Thu May  8 10:08:27 2008
Subject: [Zope] Making Zope stop storing old object versions
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2307@spi-mail2003.spitech.com>
Message-ID: <024101c8b114$fd839360$677ba8c0@DESKTOP>


<snip>
----- Original Message ----- 
From: "Sinang, Danny" <D.Sinang@spi-bpo.com>
To: "Jonathan" <dev101@magma.ca>
Cc: <zope@zope.org>
Sent: Thursday, May 08, 2008 10:03 AM
Subject: RE: [Zope] Making Zope stop storing old object versions


How do I go about replacing a Catalog using MySQL ? This is the first
time I've heard this.
</snip>

Here's the relevant chapter in the Zope Book: 
http://plope.com/Books/2_7Edition/RelationalDatabases.stx



Jonathan 

From D.Sinang at spi-bpo.com  Thu May  8 10:15:18 2008
From: D.Sinang at spi-bpo.com (Sinang, Danny)
Date: Thu May  8 10:16:52 2008
Subject: [Zope] Making Zope stop storing old object versions
In-Reply-To: <024101c8b114$fd839360$677ba8c0@DESKTOP>
Message-ID: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2310@spi-mail2003.spitech.com>

Thanks Jonathan. But I thought you were referring to a way to make
Catalog writes transparently write to MySQL.

The article you're pointing to teaches how to use use ZSQLMethods, which
we're already doing.

But the product we're using to do workflows (OpenFlow) uses the
Zcatalog. So we either rewrite it to use MySQL, or we find a way to make
Catalogs write to MySQL.

Regards,
Danny 

-----Original Message-----
From: Jonathan [mailto:dev101@magma.ca] 
Sent: Thursday, May 08, 2008 10:08 PM
To: Sinang, Danny
Cc: zope@zope.org
Subject: Re: [Zope] Making Zope stop storing old object versions


<snip>
----- Original Message -----
From: "Sinang, Danny" <D.Sinang@spi-bpo.com>
To: "Jonathan" <dev101@magma.ca>
Cc: <zope@zope.org>
Sent: Thursday, May 08, 2008 10:03 AM
Subject: RE: [Zope] Making Zope stop storing old object versions


How do I go about replacing a Catalog using MySQL ? This is the first
time I've heard this.
</snip>

Here's the relevant chapter in the Zope Book: 
http://plope.com/Books/2_7Edition/RelationalDatabases.stx



Jonathan 

From simon at joyful.com  Thu May  8 23:45:27 2008
From: simon at joyful.com (Simon Michael)
Date: Thu May  8 23:45:47 2008
Subject: [Zope] zope 2 wiki updated
Message-ID: <g00hcn$256$1@ger.gmane.org>

Good day all.. http://wiki.zope.org/zope2 has just received a pretty 
major cleanup (I can tell by my inbox, this was about a 400-er.) Your 
critique, help and improvements are welcome.

Best - Simon

From martijn at fourdigits.nl  Fri May  9 05:30:43 2008
From: martijn at fourdigits.nl (Martijn Jacobs)
Date: Fri May  9 05:30:56 2008
Subject: [Zope] Question about adapters
Message-ID: <482419C3.10200@fourdigits.nl>


Hello guys.

A question about adapters : I've written two adapters wich implement my 
own interface IMyAdapter. The first adapter adapts 
CMFCore.interfaces._content.IFolderish objects and the other one 
CMFCore.interfaces._content.IContentish objects.

One of the base classes of the IFolderish objects also implement the 
IContentish interface, so when I adapt folderish objects with 
IMyAdapter(object), the adapter registered for the IContentish objects 
always "wins". (and if it's a folderish object I want the adapter for 
IFolderish to jump in).

In other words : How can I influence the order of adapter lookup?

I'm feeling there should be a simple solution for this (applying a 
marker interface to folderish objects?), so I hope someone can point me 
in the right direction.


Kind regards,



Martijn.

-- 
Martijn Jacobs
Four Digits, Internet Solutions

a: Willemsplein 15-1 6811 KB Arnhem NL 
kvk: 091621370000 | btw: 8161.22.234.B01
e-mail: martijn@fourdigits.nl | web: http://www.fourdigits.nl
tel: +31 (0)26 44 22 700 | fax: +31 (0)84 22 06 117 

From zopelist at betabug.ch  Fri May  9 05:46:50 2008
From: zopelist at betabug.ch (Sascha Welter)
Date: Fri May  9 05:46:50 2008
Subject: [Zope] Making Zope stop storing old object versions
In-Reply-To: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2134@spi-mail2003.spitech.com>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2134@spi-mail2003.spitech.com>
Message-ID: <20080509094650.GA8209@betabug.ch>

(Thu, May 08, 2008 at 08:08:28AM +0800) Sinang, Danny
wrote/schrieb/egrapse:
> We've got lots of Catalog reads and writes and our Data.fs is growing
> at an immense rate.

Sometimes you have to look at your product code and figure out what it's
exactly doing. For example many times not very optimal products will
change all the instance when only a small attribute changes, resulting
in a zodb write that is much larger then really needed.

I documented an example of a hunt for a case like that in
"Quasi-Normal in Numbers": http://betabug.ch/blogs/ch-athens/708

Regards,

Sascha

From tino at wildenhain.de  Fri May  9 08:56:44 2008
From: tino at wildenhain.de (Tino Wildenhain)
Date: Fri May  9 08:56:53 2008
Subject: [Zope] Making Zope stop storing old object versions
In-Reply-To: <001c01c8b0fc$f344d200$677ba8c0@DESKTOP>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2134@spi-mail2003.spitech.com>	<48226B7E.2000409@wildenhain.de>
	<001c01c8b0fc$f344d200$677ba8c0@DESKTOP>
Message-ID: <48244A0C.7030003@wildenhain.de>

Hi,

Jonathan wrote:
>>> Sinang, Danny wrote:
>>> Hello,
...
>> You can also try some of the other storages like relstorage (where it
>> depends on the RDBMS how it handles writes)
> 
> Alternatively, you could just replace your Catalog with an external db 
> such as MySQL (using zmysql database connections).

well if MySQL is the right tool and does not introduce other problems.
It should be pointed out that along the powerful open source DBs there
is for example also postgresql which is extreamly good at handling 
concurrent loads.

Cheers
Tino
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3241 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mail.zope.org/pipermail/zope/attachments/20080509/83d4c2eb/smime.bin
From mdriscoll at co.marshall.ia.us  Fri May  9 09:31:29 2008
From: mdriscoll at co.marshall.ia.us (Mike Driscoll)
Date: Fri May  9 09:31:40 2008
Subject: [Zope] Weird SQL Relay conversion error in Zope
In-Reply-To: <8251249.17401210339802203.JavaMail.root@zimbra>
Message-ID: <7758798.17461210339889756.JavaMail.root@zimbra>

Hi,

I am working on a web app that interfaces with Microsoft SQL Server from Ubuntu through SQL Relay. I will be getting data from multiple databases. The issue seems to be that either Zope or SQL Relay cannot handle the Decimal type if the value is a null. If I do a query like this:

SELECT * from myDatabase

I get the following:

<error>

Site Error

An error was encountered while publishing this resource.

InvalidOperation
Sorry, a site error occurred.

Traceback (innermost last):

    * Module ZPublisher.Publish, line 194, in publish_module_standard
    * Module Products.PlacelessTranslationService.PatchStringIO, line 34, in new_publish
    * Module ZPublisher.Publish, line 146, in publish
    * Module Zope2.App.startup, line 222, in zpublisher_exception_hook
    * Module ZPublisher.Publish, line 115, in publish
    * Module ZPublisher.mapply, line 88, in mapply
    * Module ZPublisher.Publish, line 41, in call_object
    * Module Shared.DC.ZRDB.Connection, line 115, in manage_test
    * Module Products.ZSQLRelayDA.db, line 86, in query
    * Module decimal, line 571, in __new__
    * Module decimal, line 2267, in _raise_error

InvalidOperation: (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: '@@plone')

</error>

If I change it to "SELECT empid, reg from myDatabase" it will work as long as reg is not a NULL. I can get it to work if I use "isnull" functions on all my decimal type fields, but I would like to avoid that. Is this a known issue with a workaround?

I am using Zope 2.9.4-final on Ubuntu with SQL Relay 0.39.4. 

Mike Driscoll
Applications Specialist
MCIS - Technology Center
From David.Bear at asu.edu  Fri May  9 12:05:06 2008
From: David.Bear at asu.edu (David Bear)
Date: Fri May  9 12:05:07 2008
Subject: [Zope] first attempt to use RELSTORAGE
Message-ID: <1d1a54bf0805090905r2153e9c9ife73ac5f4a49490c@mail.gmail.com>

I installed relstorage using the instructions for easy_setup, i.e.
'easy_install RelStorage'. I realized as I read through more of the
instructions that this likely put the relstorage modules in my python home
rather than in my zope software home. I don't know if this would have caused
the error I am seeing.

However, after reading more through the instructions, and watching
easy_install do its magic, I reached a point where I make a new zope
instance, then when in to that instance home and modified the zope.conf  to
use relstorage instead of filestorage, I was ready to start my new instance.
I was disappointed to have a traceback just running zopectl. I ran
$INSTANCE_HOME/bin/zopectl

and got the following traceback:

Traceback (most recent call last):
 File "/servers/webenv1/lib/python/Zope2/Startup/zopectl.py", line 329, in ?
   main()
 File "/servers/webenv1/lib/python/Zope2/Startup/zopectl.py", line 287, in
main
   options.realize(args)
 File "/servers/webenv1/lib/python/Zope2/Startup/zopectl.py", line 95, in
realize
   ZDOptions.realize(self, *args, **kw)
 File
"/servers/webenv1/lib/python2.4/site-packages/zdaemon-2.0.2-py2.4.egg/zdaemon/zdoptions.py",
line 279, in realize
   self.load_schema()
 File
"/servers/webenv1/lib/python2.4/site-packages/zdaemon-2.0.2-py2.4.egg/zdaemon/zdoptions.py",
line 327, in load_schema
   self.schema = ZConfig.loadSchema(self.schemafile)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/loader.py",
line 32, in loadSchema
   return SchemaLoader().loadURL(url)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/loader.py",
line 66, in loadURL
   return self.loadResource(r)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/loader.py",
line 187, in loadResource
   schema = ZConfig.schema.parseResource(resource, self)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/schema.py",
line 27, in parseResource
   xml.sax.parse(resource.file, parser)
 File "/servers/webenv1//lib/python2.4/xml/sax/__init__.py", line 33, in
parse
   parser.parse(source)
 File "/servers/webenv1//lib/python2.4/xml/sax/expatreader.py", line 107, in
parse
   xmlreader.IncrementalParser.parse(self, source)
 File "/servers/webenv1//lib/python2.4/xml/sax/xmlreader.py", line 123, in
parse
   self.feed(buffer)
 File "/servers/webenv1//lib/python2.4/xml/sax/expatreader.py", line 207, in
feed
   self._parser.Parse(data, isFinal)
 File "/servers/webenv1//lib/python2.4/xml/sax/expatreader.py", line 300, in
start_element
   self._cont_handler.startElement(name, AttributesImpl(attrs))
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/schema.py",
line 99, in startElement
   getattr(self, "start_" + name)(attrs)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/schema.py",
line 475, in start_schema
   keytype, valuetype, datatype = self.get_sect_typeinfo(attrs)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/schema.py",
line 201, in get_sect_typeinfo
   datatype = self.get_datatype(attrs, "datatype", "null", base)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/schema.py",
line 194, in get_datatype
   return self._registry.get(dtname)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/datatypes.py",
line 398, in get
   t = self.search(name)
 File
"/servers/webenv1/lib/python2.4/site-packages/ZConfig-2.5.1-py2.4.egg/ZConfig/datatypes.py",
line 423, in search
   package = __import__(n, g, g, component)
 File "/servers/webenv1/lib/python/Zope2/Startup/datatypes.py", line 21, in
?
   import OFS.Uninstalled
 File "/servers/webenv1/lib/python/OFS/Uninstalled.py", line 20, in ?
   import SimpleItem, Globals, Acquisition
 File "/servers/webenv1/lib/python/OFS/SimpleItem.py", line 26, in ?
   import AccessControl.Role, AccessControl.Owned, App.Common
 File "/servers/webenv1/lib/python/AccessControl/__init__.py", line 17, in ?
   from Implementation import setImplementation
 File "/servers/webenv1/lib/python/AccessControl/Implementation.py", line
98, in ?
   setImplementation("C")
 File "/servers/webenv1/lib/python/AccessControl/Implementation.py", line
51, in setImplementation
   from AccessControl import ImplC as impl
 File "/servers/webenv1/lib/python/AccessControl/ImplC.py", line 18, in ?
   from cAccessControl import rolesForPermissionOn, \
 File "/servers/webenv1/lib/python/AccessControl/SimpleObjectPolicies.py",
line 82, in ?
   from DocumentTemplate.DT_Util import TemplateDict
 File "/servers/webenv1/lib/python/DocumentTemplate/__init__.py", line 21,
in ?
   from DocumentTemplate import String, File, HTML, HTMLDefault, HTMLFile
 File "/servers/webenv1/lib/python/DocumentTemplate/DocumentTemplate.py",
line 112, in ?
   from DT_String import String, File
 File "/servers/webenv1/lib/python/DocumentTemplate/DT_String.py", line 19,
in ?
   from DT_Util import ParseError, InstanceDict, TemplateDict,
render_blocks, str
 File "/servers/webenv1/lib/python/DocumentTemplate/DT_Util.py", line 67, in
?
   from ZPublisher.TaintedString import TaintedString
 File "/servers/webenv1/lib/python/ZPublisher/__init__.py", line 24, in ?
   from Publish import publish_module, Retry
 File "/servers/webenv1/lib/python/ZPublisher/Publish.py", line 24, in ?
   from zope.app.publication.browser import setDefaultSkin
 File "/servers/webenv1/lib/python/zope/app/publication/browser.py", line
33, in ?
   from zope.app.publication.http import BaseHTTPPublication
 File "/servers/webenv1/lib/python/zope/app/publication/http.py", line 27,
in ?
   from zope.app.publication.zopepublication import ZopePublication
 File "/servers/webenv1/lib/python/zope/app/publication/zopepublication.py",
line 46, in ?
   from zope.app.security.principalregistry import principalRegistry as
prin_reg
 File "/servers/webenv1/lib/python/zope/app/security/principalregistry.py",
line 20, in ?
   from zope.app.authentication.interfaces import IPasswordManager
 File "/servers/webenv1/lib/python/zope/app/authentication/__init__.py",
line 20, in ?
   from zope.app.authentication.authentication import
PluggableAuthentication
 File
"/servers/webenv1/lib/python/zope/app/authentication/authentication.py",
line 25, in ?
   from zope.app.component.site import SiteManagementFolder
 File "/servers/webenv1/lib/python/zope/app/component/site.py", line 35, in
?
   from zope.app.component import adapter
 File "/servers/webenv1/lib/python/zope/app/component/adapter.py", line 28,
in ?
   class LocalSurrogate(zope.interface.adapter.Surrogate):
AttributeError: 'module' object has no attribute 'Surrogate'


for more information, there is another zope instance running on this machine
however there is only 1 zope_home. Furthermore, what I couldn't discern from
the install instructions for relstorage was configuration for postgresql.
Since the instructions were silent, I assumed I at least needed a pg user
for zope to connect as, and a database to contain the zodb. So I created
these in pg so the dsn in zope.conf would at least have something valid to
connect to.

Any adivce?

I think I will need to back out of what easy_install did to my pythonhome --
is there an easy way to undo what easy_install did ?


-- 
David Bear
College of Public Programs at ASU
602-464-0424
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20080509/b8f29736/attachment.html
From anand.prabhakar.patil at gmail.com  Fri May  9 12:05:17 2008
From: anand.prabhakar.patil at gmail.com (Anand Patil)
Date: Fri May  9 12:05:17 2008
Subject: [Zope] Job opening: Malaria Atlas Project, Oxford University
Message-ID: <2bc7a5a50805090905o4e1e515docb14beafc721842a@mail.gmail.com>

DEPARTMENT OF ZOOLOGY

Web developer
Grade 7: Salary range:?26,666 - ?32,796
Three year post

A web developer is required to join the Malaria Atlas Project (MAP,
http://www.map.ox.ac.uk). The job will be to lead in the improvement of the
on-line presence and thus public engagement activities of MAP. Specific
responsibilities will be to implement an online database of epidemiological
information for MAP members and ultimately the wider malaria public health
community. The successful candidate will also be required to develop the
website functionality to display MAP products through Google Maps and Google
Earth. It will be necessary to demonstrate a portfolio of skills and
existing project experience in these areas.

Further particulars can be downloaded from
http://www.zoo.ox.ac.uk/jobs/current vacancies or are available from
the Personnel Office, Department of
Zoology, Tinbergen Building, South Parks Road, Oxford OX1 3PS. E-mail;
recruit@zoo.ox.ac.uk. Applications, including curriculum vitae, giving
contact details of three referees should be sent to the above address,
quoting reference AT08016. The closing date for applications is 06 June
2008.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20080509/deaa5499/attachment.html
From mj at zopatista.com  Fri May  9 12:44:47 2008
From: mj at zopatista.com (Martijn Pieters)
Date: Fri May  9 12:44:48 2008
Subject: [Zope] first attempt to use RELSTORAGE
In-Reply-To: <1d1a54bf0805090905r2153e9c9ife73ac5f4a49490c@mail.gmail.com>
References: <1d1a54bf0805090905r2153e9c9ife73ac5f4a49490c@mail.gmail.com>
Message-ID: <21a6ef160805090944u3ca5317en57f127ac8461a3d9@mail.gmail.com>

On Fri, May 9, 2008 at 6:05 PM, David Bear <David.Bear@asu.edu> wrote:
> I installed relstorage using the instructions for easy_setup, i.e.
> 'easy_install RelStorage'. I realized as I read through more of the
> instructions that this likely put the relstorage modules in my python home
> rather than in my zope software home. I don't know if this would have caused
> the error I am seeing.

Probably. RelStorage declares a dependency on ZODB, and likely an
incompatible version has been installed in your python's site-packages
directory.

Remove  the RelStorage, ZODB and any zope-related eggs from your
python site-packages directory to remove.

Better to not use easy_install for your Zope2 install, as Zope2
doesn't (yet) declare that it already *has* ZODB included. Just
install RelStorage in your instance home lib/python directory. Note
that RelStorage requires you to patch your Zope installation!

-- 
Martijn Pieters
From dieter at handshake.de  Sat May 10 15:54:49 2008
From: dieter at handshake.de (Dieter Maurer)
Date: Sat May 10 15:55:20 2008
Subject: [Zope] Turning back system clock
In-Reply-To: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C0E@spi-mail2003.spitech.com>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF1C0E@spi-mail2003.spitech.com>
Message-ID: <18469.64905.763738.109157@gargle.gargle.HOWL>

Sinang, Danny wrote at 2008-5-6 10:28 +0800:
>The system clock of our Zope 2.7 machine accidentally got set to 2009.
>We noticed this mistake 6 hours later and we decided to set back the
>clock to 2008. 
> 
>How does this affect the integrity of Data.fs ?

Formerly, it was deadly.

Nowadays, most operations will succeed but packing will no longer
be possible (until 2009).

Here some explications.

A "FileStorage" file is a list of transaction records, identified
by a transaction id, which usually is also a timestamp.
"FileStorage" requires that the transaction ids are strictly increasing.

If transactions records were written (i.e. the corresponding transactions
committed) while you have had the wrong time, then your
storage contains some transaction records with ids corresponding to
a date in 2009.
To ensure that transaction ids are increasing, all new transaction ids
are forced to be larger than all existing ones. This means, they seem
to have happened in 2009 immediately after the latest one -- all
of them within an incredibly short time span.


As packing is usually important, you need to fix your
storage before it can be used again in a normal way.
You would use the "transaction iterators" in the
"FileStorage" package to read all transactions, fix the transaction
id (if necessary) and then write it to a new storage file.



-- 
Dieter
From dieter at handshake.de  Sat May 10 15:56:22 2008
From: dieter at handshake.de (Dieter Maurer)
Date: Sat May 10 15:56:53 2008
Subject: [Zope] disappearing external methods
In-Reply-To: <200805071652.35998.garry@schoolteachers.co.uk>
References: <200805071652.35998.garry@schoolteachers.co.uk>
Message-ID: <18469.64998.620993.496942@gargle.gargle.HOWL>

Garry Saddington wrote at 2008-5-7 16:52 +0100:
>I have recently moved our Zope installation over to a centos server. Since the 
>move all the external methods periodically stop working for no apparent 
>reason and the error is returned as:
>"external method can not be called because it is none"

This means that the "ExternalMethod"s source cannot be imported.
Check that this source is where it should be (and is readable).

If this is the case, press the "save" button of the "ExternalMethod"
to see a more detailed problem report.



-- 
Dieter
From stefan at epy.co.at  Sun May 11 05:03:21 2008
From: stefan at epy.co.at (Stefan H. Holek)
Date: Sun May 11 05:03:23 2008
Subject: [Zope] first attempt to use RELSTORAGE
In-Reply-To: <1d1a54bf0805090905r2153e9c9ife73ac5f4a49490c@mail.gmail.com>
References: <1d1a54bf0805090905r2153e9c9ife73ac5f4a49490c@mail.gmail.com>
Message-ID: <F1CE7C24-455A-426E-AAC7-B4389DD11217@epy.co.at>

There is a buldout-friendly version of RS now. A sample buildout.cfg  
file has been posted to zodb-dev:
http://mail.zope.org/pipermail/zodb-dev/2008-April/011828.html

HTH,
Stefan


On 9. Mai 2008, at 18:05, David Bear wrote:

> Any adivce?

--
Anything that, in happening, causes something else to happen,
causes something else to happen.  --Douglas Adams


From D.Sinang at spi-bpo.com  Mon May 12 04:39:02 2008
From: D.Sinang at spi-bpo.com (Sinang, Danny)
Date: Mon May 12 04:39:05 2008
Subject: [Zope] Getting records X to X+20 of catalog search
Message-ID: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2718@spi-mail2003.spitech.com>

Dear All,
 
How do we get records X to X+20 of a catalog search ?
 
We're aware of the "limit" parameter of the search () function, but this
only gives us the top Z rows. 
 
If I want to show records 90 to 100, I'm trying to avoid having to fetch
records 1 to 89.
 
Regards,
Danny
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20080512/caeb7cfc/attachment.html
From lists at zopyx.com  Mon May 12 08:45:05 2008
From: lists at zopyx.com (Andreas Jung)
Date: Mon May 12 08:45:13 2008
Subject: [Zope] Getting records X to X+20 of catalog search
In-Reply-To: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2718@spi-mail2003.spitech.com>
References: <0594D1BDF04FBC4B8F29E454DDE6E4AA0DEF2718@spi-mail2003.spitech.com>
Message-ID: <C41F0B2D83E89CBE086F55FC@[192.168.0.24]>



--On 12. Mai 2008 16:39:02 +0800 "Sinang, Danny" <D.Sinang@spi-bpo.com> 
wrote:

> Dear All,
>
> How do we get records X to X+20 of a catalog search ?
>
> We're aware of the "limit" parameter of the search () function, but this
> only gives us the top Z rows.
>
> If I want to show records 90 to 100, I'm trying to avoid having to fetch
> records 1 to 89.

Catalog operations become basically expensive when you call getObject(). 
Slicing the resultset first should not be harmful.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080512/793b8e13/attachment.bin
From b311b-news at theotherbell.com  Mon May 12 21:59:28 2008
From: b311b-news at theotherbell.com (Brenda Bell)
Date: Mon May 12 22:05:08 2008
Subject: [Zope] Confused about permissions
Message-ID: <b311b-news-555B7E.01592813052008@news.gmane.org>

I have a simple use case for Zope where a group of users need to be able 
to manage content stored in external files.  I installed ExternalFile 
and it seems to be working as expected for the admin user.

I've created a folder where all ExternalFile objects reside.  I've 
defined a local role "Editor" for that folder, added a user and assigned 
that user to the Editor role.

What I can't seem to do is get the permissions right such that a user in 
the Editor role can add new External Files.

I've checked every box on the Security tab for the Editor role.  Users 
in the Editor role can view and modify existing ExternalFile's but still 
cannot add new ones.

I thought maybe the problem had to do with permissions on the 
ExternalFile product itself, but when reading the description on the 
Define Permissions tab I couldn't really figure out what it was trying 
to say:

"The first column below lists the permissions for this object. The 
second specifies the permissions that should have this permission in 
this product or ZClass. For ZClass methods, only permissions that are 
defined for the ZClass are permitted.
In general, any permissions that include operations that change (mutate) 
an object should be disabled."

What exactly does this mean?

More to the point:  What permissions do I need to assign to my Editor 
role so users can add External Files?

From garry at schoolteachers.co.uk  Tue May 13 03:14:05 2008
From: garry at schoolteachers.co.uk (Garry Saddington)
Date: Tue May 13 03:07:17 2008
Subject: [Zope] centos
Message-ID: <200805130814.05378.garry@schoolteachers.co.uk>

Does anyone have any experience running Zope on Centos 5. I have some Python 
scripts which just will not work. They work on Windows, Ubuntu, and SuSe but 
just don't do anything on Centos. I have tried various Zope 2.9.0 and 
2.9.6-final, various psycopg connectors and Postgres 8.1 and 8.3.
Here is an example script that works everywhere except Centos(excuse email 
formatting):

request=context.REQUEST
datefrom=request.datefrom
dateto=request.dateto
today = DateTime(datefrom)
end = DateTime(dateto)
auth=request.auth
year=request.year
while today <= end:
    for result in context.getstudentsbyyear(year=year):
        studentid=result.studentid
        if context.getday(today=today): #picks out non-weekend days only
               if context.checkholidays(today=today): #picks out holidays
                   pass
               else:
                   
\\\\\\\\context.insertabsence(entered=today,auth=auth,studentid=studentid)
            
    today += 1
return "<b class=red>Absences entered</b>"

Any ideas because I am getting very frustrated here.
Regards
Garry
From lists at zopyx.com  Tue May 13 03:09:36 2008
From: lists at zopyx.com (Andreas Jung)
Date: Tue May 13 03:09:41 2008
Subject: [Zope] centos
In-Reply-To: <200805130814.05378.garry@schoolteachers.co.uk>
References: <200805130814.05378.garry@schoolteachers.co.uk>
Message-ID: <588280F482A59BE132A52BA9@[192.168.0.24]>



--On 13. Mai 2008 08:14:05 +0100 Garry Saddington 
<garry@schoolteachers.co.uk> wrote:

> Does anyone have any experience running Zope on Centos 5. I have some
> Python  scripts which just will not work. They work on Windows, Ubuntu,
> and SuSe but  just don't do anything on Centos. I have tried various Zope
> 2.9.0 and  2.9.6-final, various psycopg connectors and Postgres 8.1 and
> 8.3. Here is an example script that works everywhere except Centos(excuse
> email  formatting):

Such functionality should be (and is) basically independent of the OS.
If there is an issue, debug the script step-by-step using pdb.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080513/3a72bb98/attachment.bin
From dev101 at magma.ca  Tue May 13 06:54:58 2008
From: dev101 at magma.ca (Jonathan)
Date: Tue May 13 06:54:55 2008
Subject: [Zope] centos
References: <200805130814.05378.garry@schoolteachers.co.uk>
Message-ID: <002b01c8b4e7$c9977310$677ba8c0@DESKTOP>


----- Original Message ----- 
From: "Garry Saddington" <garry@schoolteachers.co.uk>
To: <zope@zope.org>
Sent: Tuesday, May 13, 2008 3:14 AM
Subject: [Zope] centos


> Does anyone have any experience running Zope on Centos 5. I have some 
> Python
> scripts which just will not work. They work on Windows, Ubuntu, and SuSe 
> but
> just don't do anything on Centos. I have tried various Zope 2.9.0 and
> 2.9.6-final, various psycopg connectors and Postgres 8.1 and 8.3.
> Here is an example script that works everywhere except Centos(excuse email
> formatting):
>
> request=context.REQUEST
> datefrom=request.datefrom
> dateto=request.dateto
> today = DateTime(datefrom)
> end = DateTime(dateto)
> auth=request.auth
> year=request.year
> while today <= end:
>    for result in context.getstudentsbyyear(year=year):
>        studentid=result.studentid
>        if context.getday(today=today): #picks out non-weekend days only
>               if context.checkholidays(today=today): #picks out holidays
>                   pass
>               else:
>
> \\\\\\\\context.insertabsence(entered=today,auth=auth,studentid=studentid)
>
>    today += 1
> return "<b class=red>Absences entered</b>"

We run Zope 2.9.2 and 2.9.4 on centos 4.3 without any problems.  What error 
is being generated?

Jonathan 

From piotr_chamera at poczta.onet.pl  Tue May 13 07:50:33 2008
From: piotr_chamera at poczta.onet.pl (Piotr Chamera)
Date: Tue May 13 07:49:39 2008
Subject: [Zope] Strange property behaviour - why? (zope 2.10/Five)
Message-ID: <48298089.30606@poczta.onet.pl>

Hello,

I don't understand property behawiour in code bellow (I am not
experienced programmmer, so I could missed something obivious).

In example bellow: "list_items1" works as I expected - it returns
list of objects from catalog but "list_items2" fails - I get Attribute 
Error or if it finds "items" through acquisition it returns this object.

But, if I uncomment line "return [1,2,3]" in code below, then both
functions return the same value - list [1,2,3].

Search of "catalog" by acquisition messes with class attribute lookup?

<code>
class Storage (SimpleItem):
     """ see IStorage interface """

     implements(IStorage)

     title = FieldProperty(IStorage['title'])
     description = FieldProperty(IStorage['description'])

     def _get_items(self):
         """ Return list of items in storage (unsorted). """
         #return [1,2,3]
         return [x.getObject() for x in self.catalog(storage=self.title)]

     items = property(_get_items)

     def list_items1(self):
         """ test """
         return self._get_items()

     def list_items2(self):
         """ test """
         return self.items
</code>

Piotr Chamera
From suresh_vv at yahoo.com  Tue May 13 09:35:31 2008
From: suresh_vv at yahoo.com (Suresh V.)
Date: Tue May 13 09:36:52 2008
Subject: [Zope] Re: Strange property behaviour - why? (zope 2.10/Five)
In-Reply-To: <48298089.30606@poczta.onet.pl>
References: <48298089.30606@poczta.onet.pl>
Message-ID: <g0c5hc$993$1@ger.gmane.org>

Piotr Chamera wrote:

> 
>     def list_items2(self):
>         """ test """
>         return self.items

try to remove acquisition wrapper before you access items?

Suresh

From b311b-zope at theotherbell.com  Tue May 13 09:27:19 2008
From: b311b-zope at theotherbell.com (Brenda Bell)
Date: Tue May 13 10:05:01 2008
Subject: [Zope] Re: Confused about permissions
References: <b311b-news-555B7E.01592813052008@news.gmane.org>
Message-ID: <b311b-zope-38D510.13271913052008@news.gmane.org>

In article <b311b-news-555B7E.01592813052008@news.gmane.org>,
 Brenda Bell <b311b-news@theotherbell.com> 
 wrote:

> What I can't seem to do is get the permissions right such that a user in 
> the Editor role can add new External Files.
> 
> I've checked every box on the Security tab for the Editor role.  Users 
> in the Editor role can view and modify existing ExternalFile's but still 
> cannot add new ones.
> 

I enabled VerboseSecurity and I see this in my console window when I try 
to add an ExternalFile when logged in as a user who is assigned the 
Editor role:

2008-05-13 13:12:55 ERROR Zope.SiteErrorLog 
http://localhost:8090/foo/manage_addProduct/ExternalFile/manage_addObject
ViaGui
Traceback (innermost last):
  Module ZPublisher.Publish, line 110, in publish
  Module ZPublisher.BaseRequest, line 596, in traverse
  Module ZPublisher.HTTPResponse, line 713, in unauthorized
Unauthorized: <strong>You are not authorized to access this 
resource.</strong><p>
No Authorization header found.</p> 

A loop in BaseRequest seems to be falling through a loop that's 
dependent on ''__allow_groups__''.

I've tried adding a group, assigning the Editor role to the group and 
making my user a member of the group.  But I still get the same error.

What am I missing?

> I thought maybe the problem had to do with permissions on the 
> ExternalFile product itself, but when reading the description on the 
> Define Permissions tab I couldn't really figure out what it was trying 
> to say:
> 
> "The first column below lists the permissions for this object. The 
> second specifies the permissions that should have this permission in 
> this product or ZClass. For ZClass methods, only permissions that are 
> defined for the ZClass are permitted.
> In general, any permissions that include operations that change (mutate) 
> an object should be disabled."
> 
> What exactly does this mean?
> 
> More to the point:  What permissions do I need to assign to my Editor 
> role so users can add External Files?
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )

From piotr_chamera at poczta.onet.pl  Tue May 13 10:14:08 2008
From: piotr_chamera at poczta.onet.pl (Piotr Chamera)
Date: Tue May 13 10:13:16 2008
Subject: [Zope] Re: Strange property behaviour - why? (zope 2.10/Five)
In-Reply-To: <g0c5hc$993$1@ger.gmane.org>
References: <48298089.30606@poczta.onet.pl> <g0c5hc$993$1@ger.gmane.org>
Message-ID: <4829A230.9030508@poczta.onet.pl>

Suresh V. pisze:
> Piotr Chamera wrote:
>>     def list_items2(self):
>>         """ test """
>>         return self.items
> try to remove acquisition wrapper before you access items?
But this function returns no value, it fails to find "items" attribute
in self and tries to find it through acquisition :(

-- 
Piotr Chamera
From garry at schoolteachers.co.uk  Tue May 13 12:15:10 2008
From: garry at schoolteachers.co.uk (Garry Saddington)
Date: Tue May 13 12:09:35 2008
Subject: [Zope] centos
In-Reply-To: <588280F482A59BE132A52BA9@[192.168.0.24]>
References: <200805130814.05378.garry@schoolteachers.co.uk>
	<588280F482A59BE132A52BA9@[192.168.0.24]>
Message-ID: <200805131715.10648.garry@schoolteachers.co.uk>

On Tuesday 13 May 2008 08:09, Andreas Jung wrote:
> --On 13. Mai 2008 08:14:05 +0100 Garry Saddington
>
> <garry@schoolteachers.co.uk> wrote:
> > Does anyone have any experience running Zope on Centos 5. I have some
> > Python  scripts which just will not work. They work on Windows, Ubuntu,
> > and SuSe but  just don't do anything on Centos. I have tried various Zope
> > 2.9.0 and  2.9.6-final, various psycopg connectors and Postgres 8.1 and
> > 8.3. Here is an example script that works everywhere except Centos(excuse
> > email  formatting):
>
> Such functionality should be (and is) basically independent of the OS.
> If there is an issue, debug the script step-by-step using pdb.
 I have now traced it to the wrong datestyle in zope.conf - a new server and 
only half the brain awake!. It was set to US and we use international so the 
method did not do anything because the entered dates were presented wrongly 
to the database.
Thanks anyway
Garry

From tseaver at palladion.com  Wed May 14 01:29:34 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Wed May 14 01:29:32 2008
Subject: [Zope] Re: one zope instance per cpu
In-Reply-To: <1EEA8FAEE559BE49B397E42A3D42A5A70BC23CADD1@floyd.universe.holmescorp.com>
References: <1EEA8FAEE559BE49B397E42A3D42A5A70BC23CADD1@floyd.universe.holmescorp.com>
Message-ID: <482A78BE.2000608@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pat McNamee wrote:
>> Generally, the rule-of-thumb for scaling Zope on a multi-CPU box is to
>> run an appserver instance per CPU, each talking to a shared storage...
> 
> I came across the preceeding post while googling for info about the
> one zope per cpu question.
> 
> First, when you say one zope instance per cpu, do you count a quad-core
> cpu as four cpus?

Four:  the idea is to split up the processes so that their threads make
best use of the available CPUs.  Python's global interpreter lock makes
using more than one CPU within a single process problematic.

> Also, I've been (unsuccessfully) searching all over for an answer to
> the question: if you create one zope instance per cpu each talking
> to a shared zeo instance, how do the instances know about the cpus?
> Does each instance automagically use a separate cpu or what?

Modern linux kernels will tend to migrate all the processes to separate
CPUs (in steady state).

> I thank you in advance for your reply, and I hope I'm not being
> rude by emailing you instead of the Zope list. If this is rude
> please tell me and I'll post to the list.

CC'ing the list does leave the conversation open, and makes future
searches likelier to show the "whole picture".


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver@palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKni9+gerLs4ltQ4RAg9rAJ4xaklSySaGqR2bdZhxE6RVL2qOPACdEqWO
lq2zMLDM4+G4q2WuqmAO2Gk=
=D5M6
-----END PGP SIGNATURE-----
From piotr_chamera at poczta.onet.pl  Wed May 14 07:44:46 2008
From: piotr_chamera at poczta.onet.pl (Piotr Chamera)
Date: Wed May 14 07:43:57 2008
Subject: [Zope] Strange property behaviour - why? (zope 2.10/Five)
In-Reply-To: <48298089.30606@poczta.onet.pl>
References: <48298089.30606@poczta.onet.pl>
Message-ID: <482AD0AE.3090106@poczta.onet.pl>


> I don't understand property behawiour in code bellow (I am not
> experienced programmmer, so I could missed something obivious).
I have found related bug at https://bugs.launchpad.net/zope2/+bug/143756
with solution for my problem.

-- 
Piotr Chamera
From KSours at carsdirect.com  Wed May 14 10:49:07 2008
From: KSours at carsdirect.com (Sours, Kevin)
Date: Wed May 14 10:49:10 2008
Subject: [Zope] Re: one zope instance per cpu
In-Reply-To: <482A78BE.2000608@palladion.com>
References: <1EEA8FAEE559BE49B397E42A3D42A5A70BC23CADD1@floyd.universe.holmescorp.com>
	<482A78BE.2000608@palladion.com>
Message-ID: <CB5A47EDB1101E43B10429FA8456C15805714432@ibex01.carsdirect.win>

 

> Four:  the idea is to split up the processes so that their threads
make best 
> use of the available CPUs.  Python's global interpreter lock makes
using more 
> than one CPU within a single process problematic.

In the multiprocess setup, is it better to run a single thread per
process or multiple threads? Seems like multiple threads would still
prevent long running requests from tying up a processor, but the
overhead may not be worth it.  I haven't had a change to do any testing,
so I was wondering if there was a concensus.
Kevin

From tseaver at palladion.com  Wed May 14 13:08:19 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Wed May 14 13:08:28 2008
Subject: [Zope] Re: Confused about permissions
In-Reply-To: <b311b-zope-38D510.13271913052008@news.gmane.org>
References: <b311b-news-555B7E.01592813052008@news.gmane.org>
	<b311b-zope-38D510.13271913052008@news.gmane.org>
Message-ID: <482B1C83.4070403@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brenda Bell wrote:
> In article <b311b-news-555B7E.01592813052008@news.gmane.org>,
>  Brenda Bell <b311b-news@theotherbell.com> 
>  wrote:
> 
>> What I can't seem to do is get the permissions right such that a user in 
>> the Editor role can add new External Files.
>>
>> I've checked every box on the Security tab for the Editor role.  Users 
>> in the Editor role can view and modify existing ExternalFile's but still 
>> cannot add new ones.

I don't know that product.  There is likely another permission, e.g.
'Add External Files', which protects the methods used to create new
instances;  that permission is likely distinct from the permissions
required to modify or delete those instances.

> I enabled VerboseSecurity and I see this in my console window when I try 
> to add an ExternalFile when logged in as a user who is assigned the 
> Editor role:
> 
> 2008-05-13 13:12:55 ERROR Zope.SiteErrorLog 
> http://localhost:8090/foo/manage_addProduct/ExternalFile/manage_addObject
> ViaGui
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 110, in publish
>   Module ZPublisher.BaseRequest, line 596, in traverse
>   Module ZPublisher.HTTPResponse, line 713, in unauthorized
> Unauthorized: <strong>You are not authorized to access this 
> resource.</strong><p>
> No Authorization header found.</p> 

That traceback should be from the first request, which then prompts you
to log in (e.g., via basic auth).

The next request should either succeed (if the login / password you
supply matches), or show a different traceback (e.g., invalid credentials).

> A loop in BaseRequest seems to be falling through a loop that's 
> dependent on ''__allow_groups__''.

'__allow_groups__' is the name which the authentication machinery uses
to find the user folder:  it is an alias for 'acl_users', not visible in
the ZMI.

> I've tried adding a group, assigning the Editor role to the group and 
> making my user a member of the group.  But I still get the same error.
> 
> What am I missing?


Rewa.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver@palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKxyD+gerLs4ltQ4RAr4BAJ9g3VW8CKoNUsTeyOeLv0q4mYJE4QCgu6Id
PoQUteaxY9oNtgwqyNQfdHQ=
=SBFz
-----END PGP SIGNATURE-----

From tseaver at palladion.com  Wed May 14 13:14:09 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Wed May 14 13:14:22 2008
Subject: [Zope] Re: one zope instance per cpu
In-Reply-To: <CB5A47EDB1101E43B10429FA8456C15805714432@ibex01.carsdirect.win>
References: <1EEA8FAEE559BE49B397E42A3D42A5A70BC23CADD1@floyd.universe.holmescorp.com>	<482A78BE.2000608@palladion.com>
	<CB5A47EDB1101E43B10429FA8456C15805714432@ibex01.carsdirect.win>
Message-ID: <482B1DE1.9010203@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sours, Kevin wrote:
>  
> 
>> Four:  the idea is to split up the processes so that their threads
> make best 
>> use of the available CPUs.  Python's global interpreter lock makes
> using more 
>> than one CPU within a single process problematic.
> 
> In the multiprocess setup, is it better to run a single thread per
> process or multiple threads? Seems like multiple threads would still
> prevent long running requests from tying up a processor, but the
> overhead may not be worth it.  I haven't had a change to do any testing,
> so I was wondering if there was a concensus.

I would still go  with multiple threads, for the cases where the
application *does* release the GIL (e.g., making a call to a relational
database, etc.)  The tradeoff is that you hit the ceiling on RAM faster
with more threads, so if your system is RAM-constrained, you should
perhaps consider reducing the number of threads.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver@palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKx3h+gerLs4ltQ4RAmiNAJ98AhMS8+pyi3p0xOjOVMVczxQGeACguaIE
NdL21l8nX3Y3NsKyNZevJxU=
=Z6LP
-----END PGP SIGNATURE-----

From dvs at zeitbyte.com  Fri May 16 10:36:35 2008
From: dvs at zeitbyte.com (dvs)
Date: Fri May 16 10:36:35 2008
Subject: [Zope] ZODB reverts to older state after ZEO restart
Message-ID: <17264827.post@talk.nabble.com>


Recently we restarted our zeo server and immediately noticed 
that we are missing good bit of recently added content. We seemed 
to have reverted back to a specific date in the past. All content 
updates since have disappeared.

Restoring the Data.fs from a recent backup still had the same effect, the 
content all reverted back to one specific date.  In the logs it does show 
that we restored the Data.fs from a backup, restarted, and migrated to 
a newer version of Zope/Plone (Zope 2.9.4 - 2.9.5, Plone 2.5.4 - 2.5.5). 
After the update, the site was fine and was running along smoothly. 

A few days later this came up in the event.log:

2008-01-24T19:05:44 ERROR ZServer uncaptured python exception, closing
channel <ZServer.HTTPServer.zhttp_channel connected 72.249.81.93:56139 at 0x
1598d92c channel#: 45214 requests:> (socket.error:(54, 'Connection reset by
peer') [/usr/local/lib/python2.4/asynchat.py|initiate_send|219] [/usr/l
ocal/zope/295/lib/python/ZServer/medusa/http_server.py|send|417]
[/usr/local/lib/python2.4/asyncore.py|send|332])
------
2008-01-24T19:05:45 ERROR ZEO.zrpc (27931) can't decode message:
'(M\x06\xdcK\x00U\x06.reply(TC\x00\x01\x00(cOFS.Image\nP...'
------
2008-01-24T19:05:45 ERROR ZEO.zrpc.Connection(C) (127.0.0.1:9999) Error
caught in asyncore
Traceback (most recent call last):
  File "/usr/local/lib/python2.4/asyncore.py", line 69, in read
    obj.handle_read_event()
  File "/usr/local/lib/python2.4/asyncore.py", line 391, in
handle_read_event
    self.handle_read()
  File "/usr/local/zope/295/lib/python/ZEO/zrpc/smac.py", line 219, in
handle_read
    self.message_input(msg)
  File "/usr/local/zope/295/lib/python/ZEO/zrpc/connection.py", line 387, in
message_input
    msgid, flags, name, args = self.marshal.decode(message)
  File "/usr/local/zope/295/lib/python/ZEO/zrpc/marshal.py", line 50, in
decode
    return unpickler.load() # msgid, flags, name, args
MemoryError
------
2008-01-24T19:05:45 INFO ZEO.ClientStorage (27931) Disconnected from
storage: "('xxxxx.com', 9999)"
------
2008-01-24T19:05:45 INFO ZEO.ClientStorage (27931) Testing connection
<ManagedClientConnection ('72.249.81.93', 9999)>
------
2008-01-24T19:05:45 INFO ZEO.zrpc.Connection(C) (127.0.0.1:9999) received
handshake 'Z303'

------
2008-01-24T19:05:45 ERROR ZODB.Connection Couldn't load state for 0x01868c
Traceback (most recent call last):
  File "/usr/local/zope/295/lib/python/ZODB/Connection.py", line 732, in
setstate
    self._setstate(obj)
  File "/usr/local/zope/295/lib/python/ZODB/Connection.py", line 786, in
_setstate
    self._reader.setGhostState(obj, p)
  File "/usr/local/zope/295/lib/python/ZODB/serialize.py", line 604, in
setGhostState
    state = self.getState(pickle)
  File "/usr/local/zope/295/lib/python/ZODB/serialize.py", line 597, in
getState
    return unpickler.load()
  File "/usr/local/zope/295/lib/python/ZODB/serialize.py", line 471, in
_persistent_load
    return self.load_oid(reference)
  File "/usr/local/zope/295/lib/python/ZODB/serialize.py", line 537, in
load_oid
    return self._conn.get(oid)
  File "/usr/local/zope/295/lib/python/ZODB/Connection.py", line 201, in get
    p, serial = self._storage.load(oid, self._version)
  File "/usr/local/zope/295/lib/python/ZEO/ClientStorage.py", line 746, in
load
    return self.loadEx(oid, version)[:2]
  File "/usr/local/zope/295/lib/python/ZEO/ClientStorage.py", line 769, in
loadEx
    data, tid, ver = self._server.loadEx(oid, version)
  File "/usr/local/zope/295/lib/python/ZEO/ServerStub.py", line 192, in
loadEx
    return self.rpc.call("loadEx", oid, version)
  File "/usr/local/zope/295/lib/python/ZEO/zrpc/connection.py", line 531, in
call
    r_flags, r_args = self.wait(msgid)
  File "/usr/local/zope/295/lib/python/ZEO/zrpc/connection.py", line 621, in
wait
    raise DisconnectedError()
DisconnectedError
------

The persistent cache file has been saved, but looking at the archive it
seems that 
you can't restore a Data.fs from the persistent cache (please correct me on
this). Looking at 
the logs we can find what content was edited in this time frame and we can
see that it did exist at one time, but the data itself is not in any of the
backup Data.fs files we have.

This is a bit perplexing in that when we restore from any of the backups it
still reverts, what could 
cause this situation and if the data was in fact being saved, where could it
have been saved? We 
were backing up the correct Data.fs (at least, it was the Data.fs that the
zeo server was connected to)

If the zeo server was read only this would make sense but then we wouldn't
be able to make any 
content updates and it would be immediately evident.

Any suggestions or ideas on how we could recover any of this data and the
reasoning behind this happening is much appreciated. 

-- 
View this message in context: http://www.nabble.com/ZODB-reverts-to-older-state-after-ZEO-restart-tp17264827p17264827.html
Sent from the Zope - General mailing list archive at Nabble.com.

From dieter at handshake.de  Fri May 16 15:52:42 2008
From: dieter at handshake.de (Dieter Maurer)
Date: Fri May 16 15:53:05 2008
Subject: [Zope] ZODB reverts to older state after ZEO restart
In-Reply-To: <17264827.post@talk.nabble.com>
References: <17264827.post@talk.nabble.com>
Message-ID: <18477.58890.579260.974354@gargle.gargle.HOWL>

dvs wrote at 2008-5-16 07:36 -0700:
>
>Recently we restarted our zeo server and immediately noticed 
>that we are missing good bit of recently added content. We seemed 
>to have reverted back to a specific date in the past. All content 
>updates since have disappeared.

This usually means that you moved or deleted your storage file
while ZEO was running.

Under *nix, processes that have a file open when it is moved or deleted
continue to operate on this file and not a potentially new file
at the original place.
However, a newly started process will the the new state and use
such a new file.


If you are lucky, the storage file was moved and you still
know the moved to location. Then, you find all your current state
in this file. Otherwise, your chances are bad....



-- 
Dieter
From dieter at handshake.de  Fri May 16 15:56:18 2008
From: dieter at handshake.de (Dieter Maurer)
Date: Fri May 16 15:56:40 2008
Subject: [Zope] Re: Confused about permissions
In-Reply-To: <b311b-zope-38D510.13271913052008@news.gmane.org>
References: <b311b-news-555B7E.01592813052008@news.gmane.org>
	<b311b-zope-38D510.13271913052008@news.gmane.org>
Message-ID: <18477.59106.987247.922390@gargle.gargle.HOWL>

Brenda Bell wrote at 2008-5-13 13:27 +0000:
>In article <b311b-news-555B7E.01592813052008@news.gmane.org>,
> Brenda Bell <b311b-news@theotherbell.com> 
> wrote:
>
>> What I can't seem to do is get the permissions right such that a user in 
>> the Editor role can add new External Files.
>> 
>> I've checked every box on the Security tab for the Editor role.  Users 
>> in the Editor role can view and modify existing ExternalFile's but still 
>> cannot add new ones.
>> 
>
>I enabled VerboseSecurity and I see this in my console window when I try 
>to add an ExternalFile when logged in as a user who is assigned the 
>Editor role:
>
>2008-05-13 13:12:55 ERROR Zope.SiteErrorLog 
>http://localhost:8090/foo/manage_addProduct/ExternalFile/manage_addObject
>ViaGui
>Traceback (innermost last):
>  Module ZPublisher.Publish, line 110, in publish
>  Module ZPublisher.BaseRequest, line 596, in traverse
>  Module ZPublisher.HTTPResponse, line 713, in unauthorized
>Unauthorized: <strong>You are not authorized to access this 
>resource.</strong><p>
>No Authorization header found.</p> 

The "manage_addObject" requires roles for which no user folder
in context could identify an appropriate user.

You could try to find out in an interactive interpreter
(started with "bin/zopectl debug" under "*nix") which
roles are required.

      app.unrestrictedTraverse('foo/manage_addProduct/ExternalFile').manage_addObject__roles__



-- 
Dieter
From dieter at handshake.de  Fri May 16 15:59:08 2008
From: dieter at handshake.de (Dieter Maurer)
Date: Fri May 16 15:59:33 2008
Subject: [Zope] Weird SQL Relay conversion error in Zope
In-Reply-To: <7758798.17461210339889756.JavaMail.root@zimbra>
References: <8251249.17401210339802203.JavaMail.root@zimbra>
	<7758798.17461210339889756.JavaMail.root@zimbra>
Message-ID: <18477.59276.475189.896194@gargle.gargle.HOWL>

Mike Driscoll wrote at 2008-5-9 08:31 -0500:
> ...
>I am working on a web app that interfaces with Microsoft SQL Server from Ubuntu through SQL Relay. I will be getting data from multiple databases. The issue seems to be that either Zope or SQL Relay cannot handle the Decimal type if the value is a null. If I do a query like this:
> ...
>    * Module Products.ZSQLRelayDA.db, line 86, in query
>    * Module decimal, line 571, in __new__
>    * Module decimal, line 2267, in _raise_error
>
>InvalidOperation: (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: '@@plone')

Looks like a bug in "ZSQLRelayDA.db".

  Apparently, it passes SQL decimal values to Python's "decimal" module --
  even if the SQL value is "Null" (almost surely mapped to Python's "None").
  For obvious reasons, "decimal" does not like to convert "None"
  into a "decimal" value.

  "ZSQLRelayDA.db" should test for "None" and call the "decimal"
  constructor only for non "None" values.



-- 
Dieter
From mdriscoll at co.marshall.ia.us  Fri May 16 16:02:55 2008
From: mdriscoll at co.marshall.ia.us (Mike Driscoll)
Date: Fri May 16 16:02:33 2008
Subject: [Zope] Weird SQL Relay conversion error in Zope
In-Reply-To: <18477.59276.475189.896194@gargle.gargle.HOWL>
References: <8251249.17401210339802203.JavaMail.root@zimbra>	<7758798.17461210339889756.JavaMail.root@zimbra>
	<18477.59276.475189.896194@gargle.gargle.HOWL>
Message-ID: <482DE86F.7050908@co.marshall.ia.us>

Dieter,
> Mike Driscoll wrote at 2008-5-9 08:31 -0500:
>   
>> ...
>> I am working on a web app that interfaces with Microsoft SQL Server from Ubuntu through SQL Relay. I will be getting data from multiple databases. The issue seems to be that either Zope or SQL Relay cannot handle the Decimal type if the value is a null. If I do a query like this:
>> ...
>>    * Module Products.ZSQLRelayDA.db, line 86, in query
>>    * Module decimal, line 571, in __new__
>>    * Module decimal, line 2267, in _raise_error
>>
>> InvalidOperation: (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: '@@plone')
>>     
>
> Looks like a bug in "ZSQLRelayDA.db".
>
>   Apparently, it passes SQL decimal values to Python's "decimal" module --
>   even if the SQL value is "Null" (almost surely mapped to Python's "None").
>   For obvious reasons, "decimal" does not like to convert "None"
>   into a "decimal" value.
>
>   "ZSQLRelayDA.db" should test for "None" and call the "decimal"
>   constructor only for non "None" values.
>
>
>   

I suspected as much. I mentioned trying SQLAlchemy to my boss and he 
tried it to see if it could handle this issue and it does. So we'll 
probably stick with that, but thanks for the clarification.

Mike



From dvs at zeitbyte.com  Fri May 16 17:20:37 2008
From: dvs at zeitbyte.com (dvs)
Date: Fri May 16 17:20:35 2008
Subject: [Zope] ZODB reverts to older state after ZEO restart
In-Reply-To: <18477.58890.579260.974354@gargle.gargle.HOWL>
References: <17264827.post@talk.nabble.com>
	<18477.58890.579260.974354@gargle.gargle.HOWL>
Message-ID: <17284454.post@talk.nabble.com>


I believe this is the case, I found another thread with a similar problem: 

http://www.nabble.com/Missing-Content-td4174116.html


Dieter Maurer wrote:
> 
> dvs wrote at 2008-5-16 07:36 -0700:
>>
>>Recently we restarted our zeo server and immediately noticed 
>>that we are missing good bit of recently added content. We seemed 
>>to have reverted back to a specific date in the past. All content 
>>updates since have disappeared.
> 
> This usually means that you moved or deleted your storage file
> while ZEO was running.
> 
> Under *nix, processes that have a file open when it is moved or deleted
> continue to operate on this file and not a potentially new file
> at the original place.
> However, a newly started process will the the new state and use
> such a new file.
> 
> 
> If you are lucky, the storage file was moved and you still
> know the moved to location. Then, you find all your current state
> in this file. Otherwise, your chances are bad....
> 
> 
> 
> -- 
> Dieter
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
> 
> 

-- 
View this message in context: http://www.nabble.com/ZODB-reverts-to-older-state-after-ZEO-restart-tp17264827p17284454.html
Sent from the Zope - General mailing list archive at Nabble.com.

From jon.emmons at earthwavetech.com  Fri May 23 08:58:25 2008
From: jon.emmons at earthwavetech.com (Jon Emmons)
Date: Fri May 23 08:58:24 2008
Subject: [Zope] Multithreading sessions
Message-ID: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon>

Dear Zope Users group,

 

I am running zope 2.9.4 and have observed that it will not simultaneously
serve pages to my users.

 

For Example:

 

User 1 requests data that takes the MySQL server 30 seconds to retrieve

 

Meanwhile.

 

User 2 wants to log on

 

User 2 will not be served the logon page until user 1's query has finished.

 

Is this typical Zope behavior?

 

How do I get Zope to serve pages, run python scripts, etc. to multiple users
simultaneously?

 

Thanks,

 

Jon Emmons

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20080523/5a29a874/attachment.html
From lukesh at seznam.cz  Fri May 23 09:10:08 2008
From: lukesh at seznam.cz (Jaroslav Lukesh)
Date: Fri May 23 09:10:22 2008
Subject: [Zope] Multithreading sessions
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon>
Message-ID: <1c3c01c8bcd6$54825530$2b0db10a@xp>

You need to increase number of server threads and/or number of database 
connections (it mean ZoDB connections).

----- Original Message ----- 
From: Jon Emmons

User 1 requests data that takes the MySQL server 30 seconds to retrieve

Meanwhile.

User 2 wants to log on

User 2 will not be served the logon page until user 1's query has finished.

Is this typical Zope behavior?

How do I get Zope to serve pages, run python scripts, etc. to multiple users 
simultaneously?

Thanks,

Jon Emmons


From lists at zopyx.com  Fri May 23 09:18:36 2008
From: lists at zopyx.com (Andreas Jung)
Date: Fri May 23 09:22:02 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon>
Message-ID: <ABB083B32534EE2828F5CFCE@[192.168.0.24]>



--On 23. Mai 2008 08:58:25 -0400 Jon Emmons <jon.emmons@earthwavetech.com> 
wrote:



>
> Is this typical Zope behavior?

No, and it is unlikely that Zope is in charge for this behavior.

You might encounter a situation where your database locks a table or 
something like that and therefore blocking other requests.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20080523/72418216/attachment.bin
From jon.emmons at earthwavetech.com  Fri May 23 09:23:40 2008
From: jon.emmons at earthwavetech.com (Jon Emmons)
Date: Fri May 23 09:23:35 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <1c3c01c8bcd6$54825530$2b0db10a@xp>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon>
	<1c3c01c8bcd6$54825530$2b0db10a@xp>
Message-ID: <002b01c8bcd8$383eb290$5702a8c0@earthwavejon>

Thanks Jaroslav,

I had already gone into zope.conf and added the line:

zserver-threads 10

This seemed to have no effect.  Is that what you mean?
You think I should bump it higher?  100 maybe?

Jon Emmons

-----Original Message-----
From: Jaroslav Lukesh [mailto:lukesh@seznam.cz] 
Sent: Friday, May 23, 2008 9:10 AM
To: Jon Emmons; zope@zope.org
Subject: Re: [Zope] Multithreading sessions

You need to increase number of server threads and/or number of database 
connections (it mean ZoDB connections).

----- Original Message ----- 
From: Jon Emmons

User 1 requests data that takes the MySQL server 30 seconds to retrieve

Meanwhile.

User 2 wants to log on

User 2 will not be served the logon page until user 1's query has finished.

Is this typical Zope behavior?

How do I get Zope to serve pages, run python scripts, etc. to multiple users

simultaneously?

Thanks,

Jon Emmons


From jon.emmons at earthwavetech.com  Fri May 23 09:31:24 2008
From: jon.emmons at earthwavetech.com (Jon Emmons)
Date: Fri May 23 09:31:19 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <ABB083B32534EE2828F5CFCE@[192.168.0.24]>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon>
	<ABB083B32534EE2828F5CFCE@[192.168.0.24]>
Message-ID: <002c01c8bcd9$4c87ea40$5702a8c0@earthwavejon>

Thanks Andreas,

And for a very long time that is what I thought.

We have 5 database servers.  They are each multisession servers.  If a query
on any one of them is taking any time, every zope user is locked up until
that query finishes, despite the server they are accessing.

I don't think it has a single thing to do with the database access, but I
have somehow gotten my zope installation in such a mode that each user
request is served serially and there is zero parallel processing.

It's not like I've seen a lot of zope installations, and I don't know how
they behave.  Typically the pages are served so rapidly that this is not a
noticeable behavior, but our user base and databases have grown to the point
that it has become critical.

Thanks for your help.

Jon Emmons


-----Original Message-----
From: Andreas Jung [mailto:lists@zopyx.com] 
Sent: Friday, May 23, 2008 9:19 AM
To: Jon Emmons; zope@zope.org
Subject: Re: [Zope] Multithreading sessions



--On 23. Mai 2008 08:58:25 -0400 Jon Emmons <jon.emmons@earthwavetech.com> 
wrote:



>
> Is this typical Zope behavior?

No, and it is unlikely that Zope is in charge for this behavior.

You might encounter a situation where your database locks a table or 
something like that and therefore blocking other requests.

-aj

From lukesh at seznam.cz  Fri May 23 09:37:28 2008
From: lukesh at seznam.cz (Jaroslav Lukesh)
Date: Fri May 23 09:37:42 2008
Subject: [Zope] Multithreading sessions
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon><1c3c01c8bcd6$54825530$2b0db10a@xp>
	<002b01c8bcd8$383eb290$5702a8c0@earthwavejon>
Message-ID: <24ef01c8bcda$26839870$2b0db10a@xp>

Hi Jon,

I use 24 zserver-threads and I have here defined number of ZoDB connections 
(pool-size):

in your instance etc/zope.conf look for that section:

<zodb_db main>
    # Main FileStorage database
    <filestorage>
      path $INSTANCE/var/Data.fs
    </filestorage>
    mount-point /
    cache-size 1000
    pool-size 24
</zodb_db>

Please pay attention, if you have bunch of cache, your RAM shoul be 
exhausted.

Regards, JL.

----- Original Message ----- 
From: "Jon Emmons" <jon.emmons@earthwavetech.com>


> I had already gone into zope.conf and added the line:
>
> zserver-threads 10
>
> This seemed to have no effect.  Is that what you mean?
> You think I should bump it higher?  100 maybe?

From jon.emmons at earthwavetech.com  Fri May 23 09:47:05 2008
From: jon.emmons at earthwavetech.com (Jon Emmons)
Date: Fri May 23 09:47:00 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <24ef01c8bcda$26839870$2b0db10a@xp>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon><1c3c01c8bcd6$54825530$2b0db10a@xp>
	<002b01c8bcd8$383eb290$5702a8c0@earthwavejon>
	<24ef01c8bcda$26839870$2b0db10a@xp>
Message-ID: <003601c8bcdb$7e100d20$5702a8c0@earthwavejon>

Jaroslav,

OK, that looks real!

Thanks,

I am going to try it on my development server and run some tests.

Jon Emmons

-----Original Message-----
From: Jaroslav Lukesh [mailto:lukesh@seznam.cz] 
Sent: Friday, May 23, 2008 9:37 AM
To: Jon Emmons; zope@zope.org
Subject: Re: [Zope] Multithreading sessions

Hi Jon,

I use 24 zserver-threads and I have here defined number of ZoDB connections 
(pool-size):

in your instance etc/zope.conf look for that section:

<zodb_db main>
    # Main FileStorage database
    <filestorage>
      path $INSTANCE/var/Data.fs
    </filestorage>
    mount-point /
    cache-size 1000
    pool-size 24
</zodb_db>

Please pay attention, if you have bunch of cache, your RAM shoul be 
exhausted.

Regards, JL.

----- Original Message ----- 
From: "Jon Emmons" <jon.emmons@earthwavetech.com>


> I had already gone into zope.conf and added the line:
>
> zserver-threads 10
>
> This seemed to have no effect.  Is that what you mean?
> You think I should bump it higher?  100 maybe?

From jon.emmons at earthwavetech.com  Fri May 23 11:09:49 2008
From: jon.emmons at earthwavetech.com (Jon Emmons)
Date: Fri May 23 11:09:44 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <24ef01c8bcda$26839870$2b0db10a@xp>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon><1c3c01c8bcd6$54825530$2b0db10a@xp>
	<002b01c8bcd8$383eb290$5702a8c0@earthwavejon>
	<24ef01c8bcda$26839870$2b0db10a@xp>
Message-ID: <003d01c8bce7$0c31d9c0$5702a8c0@earthwavejon>

Jaroslav and everyone,

I have made the changes Jaroslav suggested and I cannot get Zope to serve
even 2 sessions simultaneously.

In our testing, one client will start a query to the data servers.  Another
will try to simply get the logon page to be served.  Zope will simply sit
there and wait until the DB returns its results for the first user before
doing anything at all for the second.  The 2nd user is not hitting the DB
server.  The python script accessing the data is waiting for the results for
the first user.  Could python be the problem?

I am running python 2.4.4.

Is it possible that I missed a flag or something when I build the Zope?

Thanks for your suggestions so far, but I think I have missed a fundamental
setup step that allows zope to multithread sessions.

If I were echoing what was being served to the clients, shouldn't I see it
interwoven?  I don't, it is always (client 1's results) then (client 2's
results)... very atomic.

Thanks again,

Jon Emmons



-----Original Message-----
From: Jaroslav Lukesh [mailto:lukesh@seznam.cz] 
Sent: Friday, May 23, 2008 9:37 AM
To: Jon Emmons; zope@zope.org
Subject: Re: [Zope] Multithreading sessions

Hi Jon,

I use 24 zserver-threads and I have here defined number of ZoDB connections 
(pool-size):

in your instance etc/zope.conf look for that section:

<zodb_db main>
    # Main FileStorage database
    <filestorage>
      path $INSTANCE/var/Data.fs
    </filestorage>
    mount-point /
    cache-size 1000
    pool-size 24
</zodb_db>

Please pay attention, if you have bunch of cache, your RAM shoul be 
exhausted.

Regards, JL.

----- Original Message ----- 
From: "Jon Emmons" <jon.emmons@earthwavetech.com>


> I had already gone into zope.conf and added the line:
>
> zserver-threads 10
>
> This seemed to have no effect.  Is that what you mean?
> You think I should bump it higher?  100 maybe?

From KSours at carsdirect.com  Fri May 23 11:48:05 2008
From: KSours at carsdirect.com (Sours, Kevin)
Date: Fri May 23 11:48:03 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <003d01c8bce7$0c31d9c0$5702a8c0@earthwavejon>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon><1c3c01c8bcd6$54825530$2b0db10a@xp><002b01c8bcd8$383eb290$5702a8c0@earthwavejon><24ef01c8bcda$26839870$2b0db10a@xp>
	<003d01c8bce7$0c31d9c0$5702a8c0@earthwavejon>
Message-ID: <CB5A47EDB1101E43B10429FA8456C1580571492E@ibex01.carsdirect.win>

What it sounds like is that Python is using a "sequential" threading
model.  I don't know about Python in particular, but some languages
implement a sequential model internally so that multithreaded programs
will run correctly (if not effeciently) in the absense of whatever
thread libraries the language uses for normal threading.

As an aside, something I learned the hard way is that Python doesn't
thread very well.  The documentation talks a lot about threading, but in
most instances only one thread can run at a given time (it should still
swap between threads, though).  I never saw any benefit to increasing
z-server-threads beyond 4 -- all the extra threads are going to do is
consume memory.  If you need get real concurrency (for example to make
use of additional processors) you need to run multiple zope processes.
Kevin

-----Original Message-----
From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of
Jon Emmons
Sent: Friday, May 23, 2008 8:10 AM
To: 'Jaroslav Lukesh'; zope@zope.org
Subject: RE: [Zope] Multithreading sessions

Jaroslav and everyone,

I have made the changes Jaroslav suggested and I cannot get Zope to
serve even 2 sessions simultaneously.

In our testing, one client will start a query to the data servers.
Another will try to simply get the logon page to be served.  Zope will
simply sit there and wait until the DB returns its results for the first
user before doing anything at all for the second.  The 2nd user is not
hitting the DB server.  The python script accessing the data is waiting
for the results for the first user.  Could python be the problem?

I am running python 2.4.4.

Is it possible that I missed a flag or something when I build the Zope?

Thanks for your suggestions so far, but I think I have missed a
fundamental setup step that allows zope to multithread sessions.

If I were echoing what was being served to the clients, shouldn't I see
it interwoven?  I don't, it is always (client 1's results) then (client
2's results)... very atomic.

Thanks again,

Jon Emmons



-----Original Message-----
From: Jaroslav Lukesh [mailto:lukesh@seznam.cz]
Sent: Friday, May 23, 2008 9:37 AM
To: Jon Emmons; zope@zope.org
Subject: Re: [Zope] Multithreading sessions

Hi Jon,

I use 24 zserver-threads and I have here defined number of ZoDB
connections
(pool-size):

in your instance etc/zope.conf look for that section:

<zodb_db main>
    # Main FileStorage database
    <filestorage>
      path $INSTANCE/var/Data.fs
    </filestorage>
    mount-point /
    cache-size 1000
    pool-size 24
</zodb_db>

Please pay attention, if you have bunch of cache, your RAM shoul be
exhausted.

Regards, JL.

----- Original Message -----
From: "Jon Emmons" <jon.emmons@earthwavetech.com>


> I had already gone into zope.conf and added the line:
>
> zserver-threads 10
>
> This seemed to have no effect.  Is that what you mean?
> You think I should bump it higher?  100 maybe?

_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )
From peter at sabaini.at  Fri May 23 12:36:26 2008
From: peter at sabaini.at (Peter Sabaini)
Date: Fri May 23 12:33:01 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <003d01c8bce7$0c31d9c0$5702a8c0@earthwavejon>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon>
	<24ef01c8bcda$26839870$2b0db10a@xp>
	<003d01c8bce7$0c31d9c0$5702a8c0@earthwavejon>
Message-ID: <200805231836.33456.peter@sabaini.at>

On Friday 23 May 2008 17:09:49 Jon Emmons wrote:
> Jaroslav and everyone,
>
> I have made the changes Jaroslav suggested and I cannot get Zope to serve
> even 2 sessions simultaneously.
>
> In our testing, one client will start a query to the data servers.  Another
> will try to simply get the logon page to be served.  Zope will simply sit
> there and wait until the DB returns its results for the first user before
> doing anything at all for the second.  The 2nd user is not hitting the DB
> server.  The python script accessing the data is waiting for the results
> for the first user.  Could python be the problem?

I very much doubt it. Zope (and Python) can easily serve many concurrent 
sessions.

You'll need to find out what exactly makes Zope hang and where it hangs. One 
option to do this is to attach with gdb to the Python process and figure it 
out from the backtrace. 
http://www.upfrontsystems.co.za/Members/jean/zope-notes/debug-spinning-zope 
talks about that a bit

Another option is to use the Deadlock Debugger product (I haven't used that 
personally though): 
http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger

hth,
peter.



> I am running python 2.4.4.
>
> Is it possible that I missed a flag or something when I build the Zope?
>
> Thanks for your suggestions so far, but I think I have missed a fundamental
> setup step that allows zope to multithread sessions.
>
> If I were echoing what was being served to the clients, shouldn't I see it
> interwoven?  I don't, it is always (client 1's results) then (client 2's
> results)... very atomic.
>
> Thanks again,
>
> Jon Emmons
>
>
>
> -----Original Message-----
> From: Jaroslav Lukesh [mailto:lukesh@seznam.cz]
> Sent: Friday, May 23, 2008 9:37 AM
> To: Jon Emmons; zope@zope.org
> Subject: Re: [Zope] Multithreading sessions
>
> Hi Jon,
>
> I use 24 zserver-threads and I have here defined number of ZoDB connections
> (pool-size):
>
> in your instance etc/zope.conf look for that section:
>
> <zodb_db main>
>     # Main FileStorage database
>     <filestorage>
>       path $INSTANCE/var/Data.fs
>     </filestorage>
>     mount-point /
>     cache-size 1000
>     pool-size 24
> </zodb_db>
>
> Please pay attention, if you have bunch of cache, your RAM shoul be
> exhausted.
>
> Regards, JL.
>
> ----- Original Message -----
> From: "Jon Emmons" <jon.emmons@earthwavetech.com>
>
> > I had already gone into zope.conf and added the line:
> >
> > zserver-threads 10
> >
> > This seemed to have no effect.  Is that what you mean?
> > You think I should bump it higher?  100 maybe?
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://mail.zope.org/pipermail/zope/attachments/20080523/370729e2/attachment.bin
From mva.led at gmail.com  Fri May 23 12:42:09 2008
From: mva.led at gmail.com (Manuel Vazquez Acosta)
Date: Fri May 23 12:41:54 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <CB5A47EDB1101E43B10429FA8456C1580571492E@ibex01.carsdirect.win>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon><1c3c01c8bcd6$54825530$2b0db10a@xp><002b01c8bcd8$383eb290$5702a8c0@earthwavejon><24ef01c8bcda$26839870$2b0db10a@xp>	<003d01c8bce7$0c31d9c0$5702a8c0@earthwavejon>
	<CB5A47EDB1101E43B10429FA8456C1580571492E@ibex01.carsdirect.win>
Message-ID: <4836F3E1.5030606@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is relevant:
http://docs.python.org/api/threads.html
http://www.pyzine.com/Issue001/Section_Articles/article_ThreadingGlobalInterpreter.html

But notice some C extensions do allow concurrency by releasing the GIL.
Most I/O operations in the standard library do that. And probably the
network library used by python to connect to MySQL does that too.

However, if I got everything Jon explained, the problems happens even
with several DB servers. :?

Best regards,
Manuel.

Sours, Kevin wrote:
> What it sounds like is that Python is using a "sequential" threading
> model.  I don't know about Python in particular, but some languages
> implement a sequential model internally so that multithreaded programs
> will run correctly (if not effeciently) in the absense of whatever
> thread libraries the language uses for normal threading.
> 
> As an aside, something I learned the hard way is that Python doesn't
> thread very well.  The documentation talks a lot about threading, but in
> most instances only one thread can run at a given time (it should still
> swap between threads, though).  I never saw any benefit to increasing
> z-server-threads beyond 4 -- all the extra threads are going to do is
> consume memory.  If you need get real concurrency (for example to make
> use of additional processors) you need to run multiple zope processes.
> Kevin
> 
> -----Original Message-----
> From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of
> Jon Emmons
> Sent: Friday, May 23, 2008 8:10 AM
> To: 'Jaroslav Lukesh'; zope@zope.org
> Subject: RE: [Zope] Multithreading sessions
> 
> Jaroslav and everyone,
> 
> I have made the changes Jaroslav suggested and I cannot get Zope to
> serve even 2 sessions simultaneously.
> 
> In our testing, one client will start a query to the data servers.
> Another will try to simply get the logon page to be served.  Zope will
> simply sit there and wait until the DB returns its results for the first
> user before doing anything at all for the second.  The 2nd user is not
> hitting the DB server.  The python script accessing the data is waiting
> for the results for the first user.  Could python be the problem?
> 
> I am running python 2.4.4.
> 
> Is it possible that I missed a flag or something when I build the Zope?
> 
> Thanks for your suggestions so far, but I think I have missed a
> fundamental setup step that allows zope to multithread sessions.
> 
> If I were echoing what was being served to the clients, shouldn't I see
> it interwoven?  I don't, it is always (client 1's results) then (client
> 2's results)... very atomic.
> 
> Thanks again,
> 
> Jon Emmons
> 
> 
> 
> -----Original Message-----
> From: Jaroslav Lukesh [mailto:lukesh@seznam.cz]
> Sent: Friday, May 23, 2008 9:37 AM
> To: Jon Emmons; zope@zope.org
> Subject: Re: [Zope] Multithreading sessions
> 
> Hi Jon,
> 
> I use 24 zserver-threads and I have here defined number of ZoDB
> connections
> (pool-size):
> 
> in your instance etc/zope.conf look for that section:
> 
> <zodb_db main>
>     # Main FileStorage database
>     <filestorage>
>       path $INSTANCE/var/Data.fs
>     </filestorage>
>     mount-point /
>     cache-size 1000
>     pool-size 24
> </zodb_db>
> 
> Please pay attention, if you have bunch of cache, your RAM shoul be
> exhausted.
> 
> Regards, JL.
> 
> ----- Original Message -----
> From: "Jon Emmons" <jon.emmons@earthwavetech.com>
> 
> 
>> I had already gone into zope.conf and added the line:
>>
>> zserver-threads 10
>>
>> This seemed to have no effect.  Is that what you mean?
>> You think I should bump it higher?  100 maybe?
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFINvPgI2zpkmcEAhgRAgdwAKCAFmjuU93kwK/N9YG1rNFIsWF0vwCffP/D
jii2wbhTczH9PhkD1VvdGYA=
=vovu
-----END PGP SIGNATURE-----
From jon.emmons at earthwavetech.com  Fri May 23 12:49:26 2008
From: jon.emmons at earthwavetech.com (Jon Emmons)
Date: Fri May 23 12:49:20 2008
Subject: [Zope] Multithreading sessions
In-Reply-To: <CB5A47EDB1101E43B10429FA8456C1580571492E@ibex01.carsdirect.win>
References: <001501c8bcd4$b1139b80$5702a8c0@earthwavejon><1c3c01c8bcd6$54825530$2b0db10a@xp><002b01c8bcd8$383eb290$5702a8c0@earthwavejon><24ef01c8bcda$26839870$2b0db10a@xp>
	<003d01c8bce7$0c31d9c0$5702a8c0@earthwavejon>
	<CB5A47EDB1101E43B10429FA8456C1580571492E@ibex01.carsdirect.win>
Message-ID: <004701c8bcf4$f7019fa0$5702a8c0@earthwavejon>

Thanks Kevin,

Another thought I had, is that Zope has only one python interpreter running,
pretty much forcing all processing to be serial by default given what you've
just said.

In the zope.conf there is a setting:
Python-check-interval
Which defaults to 500 (I assume milliseconds, no documentation to confirm)

It is supposed to check the various threads at that interval.  If the
multi-threading is "sequential" then I may be out of luck.  However, given
what you said, I should see even 1 cpu give each session thread time slices.

Just a thought; I am still trying to find a definitive answer to what is
happening, or a clear path to true multithreaded session handling.



Jon Emmons


-----Original Message-----
From: Sours, Kevin [mailto:KSours@carsdirect.com] 
Sent: Friday, May 23, 2008 11:48 AM
To: Jon Emmons; zope@zope.org
Subject: RE: [Zope] Multithreading sessions

What it sounds like is that Python is using a "sequential" threading
model.  I don't know about Python in particular, but some languages
implement a sequential model internally so that multithreaded programs
will run correctly (if not effeciently) in the absense of whatever
thread libraries the language uses for normal threading.

As an aside, something I learned the hard way is that Python doesn't
thread very well.  The documentation talks a lot about threading, but in
most instances only one thread can run at a given time (it should still
swap between threads, though).  I never saw any benefit to increasing
z-server-threads beyond 4 -- all the extra threads are going to do is
consume memory.  If you need get real concurrency (for example to make
use of additional processors) you need to run multiple zope processes.
Kevin

-----Original Message-----
From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of
Jon Emmons
Sent: Friday, May 23, 2008 8:10 AM
To: 'Jaroslav Lukesh'; zope@zope.org
Subject: RE: [Zope] Multithreading sessions

Jaroslav and everyone,

I have made the changes Jaroslav suggested and I cannot get Zope to
serve even 2 sessions simultaneously.

In our testing, one client will start a query to the data servers.
Another will try to simply get the logon page to be served.  Zope will
simply sit there and wait until the DB returns its results for the first
user before doing anything at all for the second.  The 2nd user is not
hitting the DB server.  The python script accessing the data is waiting
for the results for the first user.  Could python be the problem?

I am running python 2.4.4.

Is it possible that I missed a flag or something when I build the Zope?

Thanks for your suggestions so far, but I think I have missed a
fundamental setup step that allows zope to multithread sessions.

If I were echoing what was being served to the clients, shouldn't I see
it interwoven?  I don't, it is always (client 1's results) then (client
2's results)... very atomic.

Thank