[ZODB-Dev] ZEO, FileStorage, and Undo

[brian.r.brinegar.1]

On Thu, 22 Aug 2002, Jeremy Hylton wrote:

>   >> brinegar@purdue.edu wrote: Hello,
>   >>
>   >> We have a large FileStorage database ~5 gigs. When any of our
>   >> users click would the "Undo" tab in a management screen the
>   >> request would eventually time out and our ZopeWatch script would
>   >> restart the server.
>   >>
>   >> Well this didn't happen every time, it only happens if 20
>   >> transactions haven't occurred, or haven't occurred recently. In
>   >> either of these cases the entire ZODB is searched for
>   >> transactions.
>
> Yes.  That's unfortunately true.
>
>   >> So I wrote a patch for undoLog in FileStorage to batch by days
>   >> instead of
>   >> # of transactions. This lets us show batches of 2 days at a time
>   >> # instead
>   >> of 20 transactions. Thus we never have to search more than 2 days
>   >> worth of the database. I also MonkeyPatched the manage_UndoForm
>   >> in App.UndoSupport.
>   >>
>   >> While doing this I noticed that at the beginning of undoLog
>   >> FileStorage acquires a lock (_lock_acquire). Is this necessary if
>   >> it's only doing reads?
>
> Yes.  Every FileStorage method that reads the database needs to hold
> the lock to protect the file position.  The read-only methods call
> seek() and read() on the file object.  If two read-only methods
> executed at the same time, there's no guarantee they would read the
> data they wanted to read.
>
>   >> We have a ZEO with 3 Zeo Clients and 1 Zeo Server when someone
>   >> hits Undo the Zeo Server locks and all 3 clients sit waiting. It
>   >> appears from experience as a user that_lock_acquire locks reads
>   >> and writes.
>
> Yes.  It's really a shame that this behavior locks up the ZEO server
> and prevents anyone else from making progress.  I've discussed this a
> few times as a worry, but I've never heard complaints about it in real
> life.  Well-- make that I had never heard until now.
>
>   >> We have a system with potential for thousands of content
>   >> maintainers. It is undesirable to have our entire site die for 20
>   >> seconds everytime one of them clicks "Undo". We could disable
>   >> Undos, however the transactional database was one of the reasons
>   >> we choose Zope.
>   >>
>   >> Any suggestions? Is there a way to lock only for writes? So that
>   >> pages can still be served?
>
> I think we could change the search process so that acquire and
> released the lock for each transaction.  That might make the undo a
> lot slower, but it would also allow other threads to make progress
> while undoLog() is running.  To implement this, we'd need to modify
> FileStorage to change its lock behavior and ZEO to run undoLog() in a
> separate thread.  These both sound easy.  If I sent you a patch, would
> you want to give it a try?

I think this is a great idea. I will try to incorporate it into the
batching patch I am already working. I can implement the changes to
undoLog, however I am not sure what needs to be changed in ZEO. If any
help would be greatly appriciated.

Thanks,
-Brian Brinegar
 ECN Web Systems Developer
 Purdue University
 West Lafayette, IN