[ZODB-Dev] Fwd: Zope/Plone/Zeo Concurrency Issue
Roel Bruggink
roel at fourdigits.nl
Wed May 29 07:53:09 UTC 2013
We've seen this too. We use sticky sessions (using a cookie) as a
workaround, like David suggested.
On Wed, May 29, 2013 at 8:51 AM, Malthe Borch <mborch at gmail.com> wrote:
> It sounds like a race condition. It might be that you've got a very
> low network latency between the web server and web browser such that
> there's hardly any time for the invalidation messages to propagate.
>
> But just to be certain it's not an issue with system time, check if
> your entire cluster is time-synchronized.
>
> \malthe
>
> On 29 May 2013 08:42, David Glick (Plone) <david.glick at plone.org> wrote:
> > On 5/28/13 7:53 PM, Dylan Jay wrote:
> >
> > Hi,
> >
> > My colleague is having the issue outlined below. I thought this list
> might
> > be better to give an insight as to what is going on?
> >
> > Begin forwarded message:
> >
> > From: Adam Terrey <adam at pretaweb.com>
> > Subject: Fwd: Zope/Plone/Zeo Concurrency Issue
> > Date: 28 May 2013 5:23:33 PM AEST
> > To: Dylan Jay <djay at pretaweb.com>
> >
> >
> >
> >
> > -------- Original Message --------
> > Subject: Zope/Plone/Zeo Concurrency Issue
> > Date: Tue, 28 May 2013 16:04:21 +1000
> > From: Adam Terrey <adam at pretaweb.com>
> > To: Zope at zope.org
> >
> >
> > Hi,
> >
> > Can someone offer some insight into what might be going on here and
> perhaps
> > how I can debug the following issue?
> >
> > In Plone there is a request patten used to create content which looks
> > like...
> >
> > (A) GET request to /MySite/createObject?type_name=Document
> > Responds with a redirect to a tempory document at a location such as
> > /MySite/portal_factory/Document/document.2013-05-28.1878040976/edit
> > (B) GET
> /MySite/portal_factory/Document/document.2013-05-28.1878040976/edit
> > Responds with from to set fields in the Document
> > (C) POST request to
> > /MySite/portal_factory/Document/document.2013-05-28.1878040976/atct_edit
> > Responds with a redirect to the final location of the document such as
> > /Mysite/my-page
> > (D) GET /Mysite/my-page
> > Responds with the newly created page
> >
> > In one of our production systems we are running multiple Zope/Plone
> > instances connecting to a Zeo server. And the above patten works about
> 95%
> > of the time. However, sometimes request (D) will respond with 404 Page
> Not
> > Found. My assumptions is that request (C) and (D) are going to different
> > instances and somehow the instance handling request (D) does not yet see
> the
> > transaction with completed the page creation.
> >
> > I've checked that the transaction commit happens before response headers
> are
> > emitted for request (C) - suspecting the case that the browser handles
> the
> > redirect before the transaction is completed - but this is clearly not
> the
> > case from what i can see. The transaction is well committed before
> response
> > headers are returned.
> >
> > Versions are as follows:
> >
> > Plone 4.1.3 (4112)
> > CMF 2.2.4
> > Zope 2.13.10
> > Python 2.7.2+ (default, Oct 4 2011, 20:06:09) [GCC 4.6.1]
> >
> > I have been using the following script to recreate the issue. I seem to
> be
> > able to produce the error quicker if I put server2 under apache-bench
> load,
> > and I have been able to recreate the issue on a local copy of our
> production
> > system but it is a far more rare case.
> >
> > auth = ('admin', 'admin')
> > server1 = 'http://localhost:46101'
> > server2 = 'http://localhost:46102'
> > site = '/Plone'
> >
> > import requests
> >
> > ses = requests.Session()
> > ses.auth=auth
> >
> > for i in range(10):
> >
> > # Create tempory Object
> > create_url = server1 + site + '/createObject?type_name=Document'
> > print "GET", create_url
> > res = ses.get (create_url)
> > tempory_obj = res.url.rsplit("/", 1)[0]
> > print "tempory object:", tempory_obj
> >
> >
> > # Submit data to create object (response will
> > # be the redirect ot the ojbect's location)
> >
> > data = {
> > 'id': '',
> > 'title': 'page' + str(i) ,
> > 'form.button.save': 'Save',
> > 'form.submitted': '1',
> > "description": "some secriot",
> > "text": "some text"
> > }
> >
> > post_url = tempory_obj + "/atct_edit"
> > print "POST", post_url
> > res = ses.post(post_url, data=data, allow_redirects=False)
> >
> > redirect_location = "/" + res.headers['location'][7:].split('/',1)[1]
> > print 'redirect_location', redirect_location
> >
> >
> > # Request object from the redirect from server2
> > redirect_url = server2 + redirect_location
> > print "GET", redirect_url
> > res = ses.get(redirect_url)
> >
> > if res.status_code != 200:
> > last404 = res
> > print res;
> > break;
> >
> > print "---"
> >
> >
> > Any help would be appreciated.
> >
> >
> > I've encountered a problem like this once or twice and had some trouble
> > reproducing it consistently, but assume it's due to ZEO invalidation
> > messages not making it to the other instance before it serves request D.
> >
> > You can work around the issue using sticky sessions in your load
> balancer.
> > If you don't want users to be stuck to a particular backend indefinitely
> you
> > can set a cookie with a brief expiration in an ObjectAddedEvent handler
> and
> > use it as the basis for the stickiness.
> >
> >
> > _______________________________________________
> > For more information about ZODB, see http://zodb.org/
> >
> > ZODB-Dev mailing list - ZODB-Dev at zope.org
> > https://mail.zope.org/mailman/listinfo/zodb-dev
> >
>
>
>
> --
> Au revoir, et tous mes voeux pour un avenir plein de succès et de bonheur
> ––
>
> Malthe Borch
> mborch at gmail.com
> _______________________________________________
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list - ZODB-Dev at zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev
>
--
Roel Bruggink
http://www.fourdigits.nl/mensen/roel-bruggink
Four Digits BV
http://www.fourdigits.nl tel: +31(0)26 4422700
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20130529/a3c71a66/attachment.html>
More information about the ZODB-Dev
mailing list