[Zope] thread-safe xml request client

[Brian Lloyd]

> I realized a few parts of my product are mutable so I wrapped
> those methods
> in a thread as such:
> lock = Lock()
> class HTTPSRequester(SimpleItemWithProperties, Thread):
>     def __init__(self):
>         Thread.__init__(self)
>
> "mutable method"
> def run(self):
>     lock.acquire()
>     self.fetch()
>     lock.release()
>
>
> where fetch() is the method with mutable data.
>
> Here are my questions regarding this setup...
> 1. Is this product actually considered thread-safe now?

Depends a bit on what you mean by "thread-safe" :)

The "Thread Safety In Zope 2" that you mentioned is mostly concerned
with data integrity. By that measure, I can't tell *for sure* from the
code you've posted, but if you are working with a mutable global you
should be ok by using the thread lock.

The real issue here is the fact that it looks like you are doing a
possibly long blocking operation (an http request). The problem with
that is that you can end up with a non-responsive site, and the lock
won't help there. For ex: one thread will block making the request.
The other threads (depending on how you write the code) will *also*
all potentially block - waiting for the lock. The end result is that
all of the server threads are stuck :(

A way around this (if the model fits for your app), is to expose
the global data through an object that only updates the data at
certain intervals, and preferably does so in a way that spawns a
thread that does not interfere with app threads (IOW, use a lock
to update the data, but get it in a non-app thread).


> 2. The fetch method makes a https socket connection using
> httplib. The first
> time it runs when zope is restarted it takes a VERY long time to make the
> socket connection(like 20 -30 seconds) and usually returns an empty string
> response from the server. The processor utilization goes up to 100% the
> entire time. However, the second request takes about 1-2 seconds
> and returns
> the correct info. I have no idea why it does this and would really like to
> know why.

Dunno on that :(

> 3. Is there any way I can create a thread or process that has all of the
> modules I use opened and ready to use. I figure this might eliminate some
> overhead on loading all of the modules per execution.

> 4. I have also noticed that when a user calls this product, Zope will stop
> responding until the response is returned. I would like to have
> the product
> be able to handle multiple simultaneous requests at once.

I'd bet this is due to the "thread pile-up" mentioned above.



Brian Lloyd        brian@zope.com
V.P. Engineering   540.361.1716
Zope Corporation   http://www.zope.com