[ZODB-Dev] zeopack requirements?

[Jim Fulton]

On Dec 17, 2007, at 7:36 AM, Chris Bainbridge wrote:

> I have a ~9GB live ZEO database. Last night I tried to pack it, but
> got this error:
>
> 2007-12-17T01:27:08 (4179) Error raised in delayed method
> Traceback (most recent call last):
>  File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/ 
> StorageServer.py",
> line 1016, in run
>    result = self._method(*self._args)
>  File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/ 
> StorageServer.py",
> line 351, in _pack_impl
>    self.storage.pack(time, referencesf)
>  File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ 
> FileStorage/FileStorage.py",
> line 1352, in pack
>    opos = p.pack()
>  File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ 
> FileStorage/fspack.py",
> line 482, in pack
>    self.gc.findReachable()
>  File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ 
> FileStorage/fspack.py",
> line 228, in findReachable
>    self.findReachableAtPacktime([z64])
>  File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ 
> FileStorage/fspack.py",
> line 315, in findReachableAtPacktime
>    self.reachable[oid] = pos
>  File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ 
> fsIndex.py",
> line 84, in __setitem__
>    tree[key[6:]] = value
> MemoryError
> 2007-12-17T01:27:16 (7412) opening storage '1' using FileStorage
> 2007-12-17T01:34:25 datastore.fs truncated, possibly due to damaged
> records at 8580513351
> 2007-12-17T01:34:25 Writing truncated data from datastore.fs to  
> datastore.fs.tr0
>
> Are there any limits on the pack operation?

No, but it *does* use a fair bit of memory.

> I assume that it doesn't
> have to hold the entire DB in memory at once?

No, but it does need to hold some large data structures in memory.

> Would bringing the
> database offline and then packing it work (ie. we run out of memory
> because we have many active clients), or is there some fundamental
> constraint in the underlying code?


I doubt that taking the database offline would make much difference.   
How much memory do you have on the machine?

How much memory did the server process use before packing?

Jim

--
Jim Fulton
Zope Corporation