[Zodb-checkins] CVS: StandaloneZODB/bsddb3Storage - README:1.7.2.1

Barry Warsaw barry@wooz.org
Wed, 19 Dec 2001 14:09:17 -0500


Update of /cvs-repository/StandaloneZODB/bsddb3Storage
In directory cvs.zope.org:/tmp/cvs-serv11259

Modified Files:
      Tag: StandaloneZODB-1_0-branch
	README 
Log Message:
Lots of updates for the current state of affairs.


=== StandaloneZODB/bsddb3Storage/README 1.7 => 1.7.2.1 ===
+Berkeley (bsddb3) Storage for ZODB/Zope
 
 
 Introduction
@@ -9,6 +9,19 @@
     tables, relying on Berkeley's transaction machinery to provide
     reliability and recoverability.
 
+    WARNING: We do not consider Berkeley storage to be production
+    quality yet.  The primary outstanding issue is poor performance
+    due to the underly Berkeley database's inability to efficiently
+    handle large `blobs'.  ZODB is designed to store the pickles of
+    persistent objects, and these pickles can sometimes be quite
+    large; pickles on the order of 10's of KB or even several MB are
+    not unheard of.
+
+    We are currently working with Sleepycat to resolve this issue, but
+    as of yet have not arrived at a satisfactory solution.  Be aware
+    that if your persistent objects have pickles over 4K in size
+    (which is likely), you will experience suboptimal performance.
+
 
 Contents
 
@@ -16,10 +29,10 @@
     implementations:
 
     - Packless.py is an implementation of an undo-less, version-less
-      storage that obviates the need for packing, except in the face
-      of cyclic garbage.  It uses a reference counting garbage
-      collection strategy to clean up garbage objects in the normal
-      case.  This storage was released in 1.0 beta 1.
+      storage that obviates the need for packing, except in the
+      presence of cyclic garbage.  It uses a reference counting
+      garbage collection strategy to clean up garbage objects in the
+      normal case.  This storage was released in 1.0 beta 1.
 
     - Minimal.py is a new implementation of an undo-less, version-less
       storage, however it currently only does reference counting
@@ -28,7 +41,7 @@
       storage, and provides a more robust temporary commit log, this
       is the wave of the future.  Because of resource constraints
       though, it is unlikely that Minimal will get much attention for
-      the 1.0 release.
+      the 1.0 final release.
 
     - Full.py is a complete storage implementation, supporting undo,
       versions, and automatic reference counting garbage collection.
@@ -44,9 +57,9 @@
       http://www.zope.org/Wikis/ZODB/TransactionalUndo
 
       Old style undo is /not/ supported in the Full.py storage, so
-      unless you have a newer version of ZODB, you will not be able to
-      perform undos.  We will not be adding old style undo support to
-      Full.py storage.
+      unless you have a newer version of Zope (2.5 should be fine),
+      you will not be able to perform undos.  We will not be adding
+      old style undo support to Full.py storage.
 
       As of 1.0 beta 3, Full storage also supports application level
       conflict resolution.  This is explained at
@@ -58,8 +71,7 @@
 
     The Full and Minimal storages have been tested with Python 2.1.1
     but should work for any Python 2.x version.  It may or may not
-    work with Python 1.5.2.  It may or may not work with Python 2.2,
-    but it will be tested against Python 2.2 before the final release.
+    work with Python 1.5.2.  It should work well with Python 2.2.
 
     It's possible they work with Python 1.5.2, but it's also not
     likely that Python 1.5.2 will be explicitly supported.  Full
@@ -74,13 +86,15 @@
 Requirements
 
     You must install Sleepcat BerkeleyDB and Robin Dunn's PyBSDDB
-    package separately.  As of this writing, BerkeleyDB is at version
-    3.3.11 and PyBSDDB is at 3.3.1.  It is highly recommended that you
-    use at least these versions of the package, as we've seen locking
-    (and other) problems with earlier releases.
+    package separately.  The Berkeley storages are known to work with
+    BerkeleyDB 3.3.11 and PyBSDDB 3.3.1.  As of this date, Sleepycat
+    has released BerkeleyDB 4.0.14 but PyBSDDB still doesn't
+    officially support it.  We do not recommend you use any version of
+    BerkeleyDB before 3.3.11 as we've seen locking (and other)
+    problems with earlier releases.
 
     To obtain the latest source release of BerkeleyDB, see the
-    SleepyCat site:
+    Sleepycat site:
 
     http://www.sleepycat.com
 
@@ -103,12 +117,25 @@
     the BerkeleyDB patch and reinstall BerkeleyDB, and then re-run the
     PyBSDDB tests.
 
+    Here's the build command I've used for the PyBSDDB distribution.
+    This assumes a default installation of BerkeleyDB 3.3.11.
+
+    % python setup.py build_ext --inplace --berkeley-db=/usr/local/BerkeleyDB.3.3/ --lflags="-Xlinker -rpath -Xlinker /usr/local/BerkeleyDB.3.3/lib"
+
     When you can run the tests which ship with PyBSDDB, you'll know
     you've been successful at both BerkeleyDB and PyBSDDB
     installation.
 
 
-Installing bsddb3Storage
+Installing bsddb3Storage as part of StandaloneZODB
+
+    You shouldnot need to do anything special to install bsddb3Storage
+    if you are installing it as part of the StandaloneZODB
+    distribution.  If you've received bsddb3Storage as a separate
+    package, then read on for installation instructions.
+
+
+Installing bsddb3Storage as a separate package
 
     The bsddb3Storage is distributed as a Python distutils package, so
     the simplest thing to do is to use distutils to install it:
@@ -128,7 +155,7 @@
     directions.
 
 
-Using bsddb3Storage
+Using bsddb3Storage with Zope
 
     By default, Zope uses a FileStorage to hold ZODB data.  To tell
     Zope to use an alternate storage such as Packless, you need to set
@@ -161,7 +188,7 @@
     remotely mounted filesystems such as NFS.
 
 
-Use with ZEO
+Using bsddb3Storage with ZEO
 
     The Berkeley storages are compatible with ZEO.  For general
     information on how to use alternate storage implementations with
@@ -193,7 +220,7 @@
     try/finally clause).
 
 
-BerkeleyDB Files
+BerkeleyDB files
 
     After Zope is started with one of the Berkeley storages, you will
     see a number of different types of files in your BerkeleyDB
@@ -227,19 +254,18 @@
     get to it soon).
 
 
-BerkeleyDB Log Files
+BerkeleyDB log files
 
     BerkeleyDB is a transactional database system.  In order to
     maintain transactional integrity, BerkeleyDB writes data to "log
     files" before the data is committed.  These log files live in the
-    BerkeleyDB "environment" directory unless you take steps to
+    BerkeleyDB environment directory unless you take steps to
     configure your BerkeleyDB environment differently.  BerkeleyDB log
-    files can become quite large, as well, so it may be necessary to
-    place them on a separate partition with lots of free disk space.
-    The log file directory can be changed by creating a file named
-    'DB_CONFIG' in the BerkeleyStorage "environment" directory you've
-    chosen within 'custom_zodb.py', customizing the following
-    content:
+    files can become quite large, so it may be necessary to place them
+    on a separate partition with lots of free disk space.  The log
+    file directory can be changed by creating a file named 'DB_CONFIG'
+    in the BerkeleyStorage environment directory you've chosen
+    (e.g. within custom_zodb.py), customizing the following content:
 
 	set_lg_dir /the/path/to/the/log/file/directory
 
@@ -247,7 +273,7 @@
     your actual database files will still be kept in the directory
     specified by the "env" setting of your custom_zodb.py; only the
     BerkeleyDB log files will be written to the directory you specify
-    in 'DB_CONFIG'.
+    in DB_CONFIG.
 
     Redirecting log files to a directory other than your environment
     directory may improve recoverability in the case of BerkeleyDB
@@ -268,7 +294,7 @@
     But be sure to read the db_archive manpages first!
 
 
-Setting BerkeleyDB Maximum Locks
+Setting BerkeleyDB maximum locks
 
     ZODB transactions can be of almost arbitrary sizes (actually, they
     "top out" at a total size of 2GB).  BerkeleyDB is configured to
@@ -284,7 +310,7 @@
     BerkeleyDB locking.
 
     To manually size locking, create (or edit) the file DB_CONFIG
-    within the BerkeleyDB "environment" directory you've chosen in
+    within the BerkeleyDB environment directory you've chosen in
     custom_zodb.py, adding the following directives to the DB_CONFIG
     file:
 
@@ -292,9 +318,9 @@
         set_lk_max_objects 500
         set_lk_max_lockers 3
 
-    Change the integers as necessary.  When one of the Berkeley
-    storages starts up, the Berkeley directives supplied in DB_CONFIG
-    will override the defaults.
+    Change the values as necessary.  When one of the Berkeley storages
+    starts up, the Berkeley directives supplied in DB_CONFIG will
+    override the defaults.
 
     Precision-sizing BerkeleyDB locking is a site-dependent task.
     Sleepycat recommends that you run the "db_stat -c" command against
@@ -332,7 +358,7 @@
     bsddb3Storage databases all use BTree access method.
 
 
-Archival and Maintenance
+Archival and maintenance
 
     Log file rotation for Berkeley DB is closely related to database
     archival.
@@ -368,17 +394,18 @@
     the above-referenced link regarding archival for more information.
 
 
-Disaster Recovery
+Disaster recovery
 
     To recover from an out-of-disk-space error on the log file
     partition, or another recoverable failure which causes the storage
     to raise a fatal exception, you may need to use the BerkeleyDB
     "db_recover" executable.  For more information, see the BerkeleyDB
     documentation at
-    http://www.sleepycat.com/docs/ref/transapp/recovery.html.
+
+	http://www.sleepycat.com/docs/ref/transapp/recovery.html
 
 
-BerkeleyDB Temp Files
+BerkeleyDB temporary files
 
     BerkeleyDB creates temporary files in the directory referenced by
     the $TMPDIR environment variable.  If you do not have a $TMPDIR
@@ -393,13 +420,14 @@
 Linux 2GB Limit
 
     BerkeleyDB is effected by the 2GB single-file-size limit on 32-bit
-    Linux ext2-based systems.  The Berkeley storage "pickle" database
+    Linux ext2-based systems.  The Berkeley storage pickle database
     (by default named "zodb_pickle"), which holds the bulk of the data
     for the Berkeley storages is particularly susceptible to large
     growth.  If you notice that this file's size (or any other
     Berkeley storage-related file) is nearing 2GB, you'll need to move
     your BerkeleyDB environment to a filesystem which supports > 2GB
-    files.
+    files.  If you enable largefile support for your filesystem, you
+    should be able to create much larger databases.
 
     IMPORTANT NOTE: If any of your BerkeleyDB files reaches the 2GB
     limit before you notice the failure situation, you will most
@@ -415,7 +443,7 @@
 
     Information about ZODB in general is kept on the ZODB Wiki at
 
-	http://www.zope.org/Wikis/ZODB/FrontPage
+	http://www.zope.org/Wikis/ZODB
 
     Information about the Berkeley storages in particular is at