[Zodb-checkins] CVS: ZODB3/bsddb3Storage - README.txt:1.3
Barry Warsaw
barry@wooz.org
Sat, 9 Nov 2002 00:31:37 -0500
Update of /cvs-repository/ZODB3/bsddb3Storage
In directory cvs.zope.org:/tmp/cvs-serv26034
Modified Files:
README.txt
Log Message:
An updated README
=== ZODB3/bsddb3Storage/README.txt 1.2 => 1.3 ===
--- ZODB3/bsddb3Storage/README.txt:1.2 Tue Nov 5 18:07:31 2002
+++ ZODB3/bsddb3Storage/README.txt Sat Nov 9 00:31:36 2002
@@ -8,17 +8,17 @@
------------
This package contains implementations for ZODB storages based on
-Sleepycat Software's BerkeleyDB and the PyBSDDB3 extension module.
-These storages save ZODB data to a bunch of BerkeleyDB tables, relying
-on Berkeley's transaction machinery to provide reliability and
-recoverability.
+Sleepycat Software's BerkeleyDB and the PyBSDDB3 Python wrapper
+module. These storages save ZODB data to a bunch of BerkeleyDB
+tables, relying on Berkeley's transaction machinery to provide
+reliability and recoverability.
Note that the Berkeley based storages are not "set and forget". The
underlying Berkeley database technology requires maintenance, careful
system resource planning, and tuning for performance. You should have
-a good working familiarity with Berkeley DB in general before trying
-to use these storages in a production environment. It's a good idea
-to read Sleepycat's own documentation, available at
+a good working familiarity with BerkeleyDB in general before trying to
+use these storages in a production environment. It's a good idea to
+read Sleepycat's own documentation, available at
http://www.sleepycat.com
@@ -28,22 +28,22 @@
Contents
--------
-Inside the bsddb3Storage package, there are four storage
+Inside the bsddb3Storage package, there are three storage
implementations:
- Full.py is a complete storage implementation, supporting
- transactional undo, versions, application level conflict
- resolution, and automatic reference counting garbage collection.
- Packing this storage is only required in order to get rid of old
- object revisions. Full storage provides some experimental
- support for cyclic garbage collection.
+ transactional undo, versions, application level conflict resolution,
+ packing, and automatic reference counting garbage collection. You
+ must pack this storage in order to get rid of old object revisions,
+ but there is also a new "autopack" strategy which packs the storage
+ in a separate thread.
- Minimal.py is an implementation of an undo-less, version-less
storage, which implements a reference counting garbage
collection strategy to remove unused objects. It is still
possible for garbage objects to persist in the face of object
- cycles, although a future release will integrate a cyclic
- garbage detector.
+ cycles, although a future release will add automatic garbage
+ collection.
- Packless.py is another, older implementation of an undo-less,
version-less, reference counting storage that also obviates the
@@ -57,57 +57,50 @@
* Packless relies on the BerkeleyDB locking subsystem, so for
very large transactions, you may run out of Berkeley locks.
-- Autopack.py is a new implementation of an undo-less,
- version-less storage that attempts to maximize write performance
- through the use of optimistic updates and sequential record
- appends. Autopack is also a prototype for an automatically
- packing storage, making it truly packless even for cyclic
- garbage collection.
+ You should consider Packless.py to be deprecated.
Compatibility
-------------
-As of this writing (23-Aug-2002) it is recommended that at least
-Python 2.1.3 or Python 2.2.1 be used with these storages. Full,
-Minimal, Autopack storages should also work with Python 2.3 (as yet
-unreleased).
+As of this writing (09-Nov-2002) it is recommended that at least
+Python 2.1.3 or Python 2.2.2 be used with these storages. Full and
+Minimal storages should also work with Python 2.3 (as yet unreleased).
Some testing has been conducted with both Zope 2.5.1, Zope 2.6, and
the Zope 3 code base. These storages have primarily been tested on
Linux.
It is recommended that you use at least BerkeleyDB 4.0.14 and PyBSDDB
-3.4.1. Earlier versions of both packages had bugs that could crash or
-hang your application.
+3.4.1 or later. Earlier versions of both packages had bugs that could
+crash or hang your application. Since PyBSDDB is not yet compatible
+with BerkeleyDB 4.1.24, stick with 4.0.14.
Requirements
------------
-You must install Sleepycat BerkeleyDB and Robin Dunn's PyBSDDB package
-separately.
+You must install Sleepycat BerkeleyDB and PyBSDDB separately.
-To obtain the latest source release of BerkeleyDB, see the Sleepycat
-Software site
+To obtain the BerkeleyDB 4.0.14, see the Sleepycat Software site for
+older releases of their software::
- http://www.sleepycat.com
+ http://www.sleepycat.com/update/index.html
-To obtain the latest source release of Robin Dunn's PyBSDDB package,
-see
+To obtain the latest source release of the PyBSDDB package, see::
http://pybsddb.sourceforge.net
Install both BerkeleyDB and PyBSDDB as per the instructions that come
-with those pacakges. For BerkeleyDB, it's generally wise to accept
+with those packages. For BerkeleyDB, it's generally wise to accept
the default configure options and do a "make install" as root. This
will install BerkeleyDB in /usr/local/BerkeleyDB.4.0
Note that because Berkeley installs itself in a non-standard location,
the dynamic linker ld.so may not be able to find it. This could
result in link errors during application startup. For systems that
-support ld.so.conf, it is highly recommended that you add
-/usr/local/BerkeleyDB.4.0/lib to that file and run ldconfig. An
+support ldconfig, it is highly recommended that you add
+/usr/local/BerkeleyDB.4.0/lib to /etc/ld.so.conf and run ldconfig. An
alternative approach is given below.
PyBSDDB comes with a standard distutils-based setup script which will
@@ -116,12 +109,12 @@
If you've extended your ld.so.conf file as above, you can build
PyBSDDB like so::
- % python setup.py build_ext -i
+ % python setup.py build_ext -i --berkeley-db=/usr/local/BerkeleyDB.4.0
Otherwise, here's the build command I've used with some success for
the PyBSDDB distribution::
- % python setup.py build_ext -i --berkeley-db=/usr/local/BerkeleyDB.4.4/ --lflags="-Xlinker -rpath -Xlinker /usr/local/BerkeleyDB.4.4/lib"
+ % python setup.py build_ext -i --berkeley-db=/usr/local/BerkeleyDB.4.0/ --lflags="-Xlinker -rpath -Xlinker /usr/local/BerkeleyDB.4.0/lib"
Then install the package like so::
@@ -155,7 +148,9 @@
You can also set up some tuning paramaters in the custom_zodb.py file.
See the comments in that file and in the BerkeleyBase.py file for
-details.
+details. For better performance, you should consider at least setting
+the config.logdir to point to a directory on a different disk than the
+one your tables are stored on.
By default, the environment path is set in custom_zodb.py to a
subdirectory of your Zope's var subdirectory. You may change this to
@@ -218,6 +213,9 @@
directory regularly. BerkeleyDB supports "hot-backup". Log files
need to be archived and cleared on a regular basis (see below).
+You really want to store your database files on a file system with
+large file support. See below for details.
+
BerkeleyDB log files
--------------------
@@ -265,7 +263,7 @@
application. Getting most of these knobs at the right setting is an
art, and will be different from system to system. We're still working
on recommendations with respect to the Full storage, but for the time
-being, you should at least read the following Sleepycat pages:
+being, you should at least read the following Sleepycat pages::
http://www.sleepycat.com/docs/ref/am_conf/cachesize.html
http://www.sleepycat.com/docs/ref/am_misc/tune.html
@@ -273,7 +271,8 @@
http://www.sleepycat.com/docs/ref/transapp/throughput.html
As you read these, it will be helpful to know that the bsddb3Storage
-databases all use BTree access method.
+databases mostly use BTree access method (Full storage has one Queue
+table to support packing).
One thing we can safely say is that the default BerkeleyDB cache size
of 256KB is way too low to be useful. The Berkeley storages
@@ -300,7 +299,7 @@
It is advantageous to automate this process, perhaps by creating a
script run by "cron" that makes use of the "db_archive" executable as
per the referenced document. One strategy might be to perform the
-following sequence of operations:
+following sequence of operations::
- shut down the process which is using BerkeleyDB (Zope or the ZEO
storage server).
@@ -328,7 +327,8 @@
To recover from an out-of-disk-space error on the log file partition,
or another recoverable failure which causes the storage to raise a
fatal exception, you may need to use the BerkeleyDB "db_recover"
-executable. For more information, see the BerkeleyDB documentation at
+executable. For more information, see the BerkeleyDB documentation
+at::
http://www.sleepycat.com/docs/ref/transapp/recovery.html
@@ -360,7 +360,7 @@
distributions have large file support.
Next, you need to make sure that your Python executable has large file
-support (LFS) built in. Python 2.2.1 is automatically configured with
+support (LFS) built in. Python 2.2.2 is automatically configured with
LFS, but for Python 2.1.3 you will need to rebuild your executable
according to the instructions on this page: