[Zope3-dev] svn.zope.org tarballs?

[Jim Fulton]

I've disabled tarball access from our ViewCVS setup on svn.zope.org.
Why?  We've had some problems with the subversion database on svn.zope.org.
Periodically, we'd get the database into a state where it needs to be
recovered. Of course, it's then down and unavailable until someone can
get around to recovering it.  When it's down, attempts to use ViewCVS
would give errors like:

Traceback (most recent call last):
   File "/usr/local/viewcvs-1.0-dev/lib/viewcvs.py", line 2610, in main
     request.run_viewcvs()
   File "/usr/local/viewcvs-1.0-dev/lib/viewcvs.py", line 265, in run_viewcvs
     self.rootpath, rev)
   File "/usr/local/viewcvs-1.0-dev/lib/vclib/svn/__init__.py", line 301, in __init__
     self.youngest = fs.youngest_rev(self.fs_ptr, self.pool)
SubversionException: ('Berkeley DB error while checkpointing after Berkeley DB transaction for filesystem /svn/repos/main/db:\nDB_INCOMPLETE: Cache flush was unable to complete', 160029)

which shows that the database needs recovery.

Needless to say, I've been a bit worried about this. Kapil suggested that he
though that the tar download feature of ViewCVS might be to blame.

I finally did some checking and found that downloading a tar ball causes
ViewCVS to use massive amounts of memory. For example, to download a 3MB
tar ball of the Zope 3 trunk, ViewCVS consumed 412 MB of memory.

What's likely happening is that someone is asking for a tar ball
further up in the tree and running the server out of memory.  At that point,
at best, the process dies and leaves the database in a bad state, needing
recovery. (We've also had the server get totally wedged.)

To get this feature back, someone will need to improve the way ViewCVS
handles tar downloads for subversion.  Perhaps people who want this
feature wuld be willing to help the ViewCVS team out. :)

Jim


Jim Fulton wrote:
> Marius Gedminas wrote:
> 
>> On Thu, May 06, 2004 at 08:00:27AM -0400, Jim Fulton wrote:
>>
>>> Butch Landingin wrote:
>>>
>>>> Will the new subversion viewcvs (viewsvn?)  site (svn.zope.org) 
>>>> incorporate tarballs like
>>>> the old cvs site? I personally use the tarball URLs to get an 
>>>> updated copy of Zope3 (mainly due to
>>>
>>>
>>> Not unless someone either adds that feature to viewcvs or tells me 
>>> how to enable some
>>> feature.
>>
>>
>>
>> /etc/viewcvs/viewcvs.conf:
>>
>>   #
>>   # ViewCVS can generate tarball from a repository on the fly.
>>   #
>>   allow_tar = 1
> 
> 
> Thanks. :)
> 
> Done.
> 
> 
> Jim
> 


-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org