[Checkins] SVN: relstorage/tags/1.1c1/ tagged 1.1c1
Shane Hathaway
shane at hathawaymix.org
Sat Aug 2 20:12:34 EDT 2008
Log message for revision 89242:
tagged 1.1c1
Changed:
A relstorage/tags/1.1c1/
D relstorage/tags/1.1c1/CHANGES.txt
A relstorage/tags/1.1c1/CHANGES.txt
D relstorage/tags/1.1c1/README.txt
A relstorage/tags/1.1c1/README.txt
D relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-7-1.patch
A relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-7-1.patch
D relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch
A relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch
D relstorage/tags/1.1c1/relstorage/
A relstorage/tags/1.1c1/relstorage/
-=-
Copied: relstorage/tags/1.1c1 (from rev 88770, relstorage/branches/1.1)
Deleted: relstorage/tags/1.1c1/CHANGES.txt
===================================================================
--- relstorage/branches/1.1/CHANGES.txt 2008-07-23 18:06:37 UTC (rev 88770)
+++ relstorage/tags/1.1c1/CHANGES.txt 2008-08-03 00:12:31 UTC (rev 89242)
@@ -1,187 +0,0 @@
-
-RelStorage 1.1c1
-
-- Added optional memcache integration. This is useful when the connection
- to the relational database has high latency.
-
-- Made it possible to set the pack and memcache options in zope.conf.
-
-- Log more info when a KeyError occurs within RelStorage.
-
-
-RelStorage 1.1b2
-
-- Made the MySQL locks database-specific rather than server-wide. This is
- important for multi-database configurations.
-
-- In the PostgreSQL adapter, made the pack lock fall back to table locking
- rather than advisory locks for PostgreSQL 8.1.
-
-- Changed a query for following object references (used during packing)
- to work around a MySQL performance bug. Thanks to Anton Stonor for
- discovering this.
-
-
-RelStorage 1.1b1
-
-- Fixed the use of setup.py without setuptools. Thanks to Chris Withers.
-
-- Fixed type coercion of the transaction extension field. This fixes
- an issue with converting databases. Thanks to Kevin Smith for
- discovering this.
-
-- Added logging to the pack code to help diagnose performance issues.
-
-- Additions to the object_ref table are now periodically committed
- during pre_pack so that the work is not lost if pre_pack fails.
-
-- Modified the pack code to pack one transaction at a time and
- release the commit lock frequently. This should help large pack
- operations.
-
-- Fixed buildout-based installation of the zodbconvert script. Thanks to
- Jim Fulton.
-
-
-RelStorage 1.0.1
-
-- The speedtest script failed if run on a test database that has no tables.
- Now the script creates the tables if needed. Thanks to Flavio Coelho
- for discovering this.
-
-- Reworked the auto-reconnect logic so that applications never see
- temporary database disconnects if possible. Thanks to Rigel Di Scala
- for pointing out this issue.
-
-- Improved the log messages explaining database connection failures.
-
-- Moved poll_invalidations to the common adapter base class, reducing the
- amount of code to maintain.
-
-
-RelStorage 1.0
-
-- Added a utility for converting between storages called zodbconvert.
-
-
-RelStorage 1.0c1
-
-- The previous fix for non-ASCII characters was incorrect. Now transaction
- metadata is stored as raw bytes. A schema migration is required; see
- notes/migrate-1.0-beta.txt.
-
-- Integrated setuptools and made an egg.
-
-
-RelStorage 1.0 beta
-
-- Renamed to reflect expanding database support.
-
-- Added support for Oracle 10g.
-
-- Major overhaul with many scalability and reliability improvements,
- particularly in the area of packing.
-
-- Moved to svn.zope.org and switched to ZPL 2.1.
-
-- Made two-phase commit optional in both Oracle and PostgreSQL. They
- both use commit_lock in such a way that the commit is not likely to
- fail in the second phase.
-
-- Switched most database transaction isolation levels from serializable
- to read committed. It turns out that commit_lock already provides
- the serializability guarantees we need, so it is safe to take advantage
- of the potential speed gains. The one major exception is the load
- connection, which requires an unchanging view of the database.
-
-- Stored objects are now buffered in a database table rather than a file.
-
-- Stopped using the LISTEN and NOTIFY statements in PostgreSQL since
- they are not strictly transactional in the sense we require.
-
-- Started using a prepared statement in PostgreSQL for getting the
- newest transaction ID quickly.
-
-- Removed the code in the Oracle adapter for retrying connection attempts.
- (It is better to just reconfigure Oracle.)
-
-- Added support for MySQL 5.0.
-
-- Added the poll_interval option. It reduces the frequency of database
- polls, but it also increases the potential for conflict errors on
- servers with high write volume.
-
-- Implemented the storage iterator protocol, making it possible to copy
- transactions to and from FileStorage and other RelStorage instances.
-
-- Fixed a bug that caused OIDs to be reused after importing transactions.
- Added a corresponding test.
-
-- Made it possible to disable garbage collection during packing.
- Exposed the option in zope.conf.
-
-- Valery Suhomlinov discovered a problem with non-ASCII data in transaction
- metadata. The problem has been fixed for all supported databases.
-
-
-PGStorage 0.4
-
-- Began using the PostgreSQL LISTEN and NOTIFY statements as a shortcut
- for invalidation polling.
-
-- Removed the commit_order code. The commit_order idea was intended to
- allow concurrent commits, but that idea is a little too ambitious while
- other more important ideas are being tested. Something like it may
- come later.
-
-- Improved connection management: only one database connection is
- held continuously open per storage instance.
-
-- Reconnect to the database automatically.
-
-- Removed test mode.
-
-- Switched from using a ZODB.Connection subclass to a ZODB patch. The
- Connection class changes in subtle ways too often to subclass reliably;
- a patch is much safer.
-
-- PostgreSQL 8.1 is now a dependency because PGStorage uses two phase commit.
-
-- Fixed an undo bug. Symptom: attempting to examine the undo log revealed
- broken pickles. Cause: the extension field was not being wrapped in
- psycopg2.Binary upon insert. Solution: used psycopg2.Binary.
- Unfortunately, this doesn't fix existing transactions people have
- committed. If anyone has any data to keep, fixing the old transactions
- should be easy.
-
-- Moved from a private CVS repository to Sourceforge.
- See http://pgstorage.sourceforge.net . Also switched to the MIT license.
-
-- David Pratt added a basic getSize() implementation so that the Zope
- management interface displays an estimate of the size of the database.
-
-- Turned PGStorage into a top-level package. Python generally makes
- top-level packages easier to install.
-
-
-PGStorage 0.3
-
-- Made compatible with Zope 3, although an undo bug apparently remains.
-
-
-PGStorage 0.2
-
-- Fixed concurrent commits, which were generating deadlocks. Fixed by
- adding a special table, "commit_lock", which is used for
- synchronizing increments of commit_seq (but only at final commit.)
- If you are upgrading from version 0.1, you need to change your
- database using the 'psql' prompt:
-
- create table commit_lock ();
-
-- Added speed tests and an OpenDocument spreadsheet comparing
- FileStorage / ZEO with PGStorage. PGStorage wins at reading objects
- and writing a lot of small transactions, while FileStorage / ZEO
- wins at writing big transactions. Interestingly, they tie when
- writing a RAM disk.
-
Copied: relstorage/tags/1.1c1/CHANGES.txt (from rev 89241, relstorage/branches/1.1/CHANGES.txt)
===================================================================
--- relstorage/tags/1.1c1/CHANGES.txt (rev 0)
+++ relstorage/tags/1.1c1/CHANGES.txt 2008-08-03 00:12:31 UTC (rev 89242)
@@ -0,0 +1,202 @@
+
+Next Release
+
+- Normalized poll-invalidation patches as Solaris' patch command would not
+ accept the current format. The patches now apply with:
+ patch -d lib/python/ZODB -p0 < poll-invalidation-1-zodb-3-X-X.patch
+
+- In MySQL, Use DROP TABLE IF EXISTS instead of TRUNCATE to clear 'temp_store'
+ because:
+ - TRUNCATE has one page of caveats in the MySQL documentation.
+ - TEMPORARY TABLEs have half a page of caveats when it comes to
+ replication.
+ - The end result is that 'temp_store' may not exist on the
+ replication slave at the exact same time(s) it exists on the
+ master.
+
+RelStorage 1.1c1
+
+- Added optional memcache integration. This is useful when the connection
+ to the relational database has high latency.
+
+- Made it possible to set the pack and memcache options in zope.conf.
+
+- Log more info when a KeyError occurs within RelStorage.
+
+
+RelStorage 1.1b2
+
+- Made the MySQL locks database-specific rather than server-wide. This is
+ important for multi-database configurations.
+
+- In the PostgreSQL adapter, made the pack lock fall back to table locking
+ rather than advisory locks for PostgreSQL 8.1.
+
+- Changed a query for following object references (used during packing)
+ to work around a MySQL performance bug. Thanks to Anton Stonor for
+ discovering this.
+
+
+RelStorage 1.1b1
+
+- Fixed the use of setup.py without setuptools. Thanks to Chris Withers.
+
+- Fixed type coercion of the transaction extension field. This fixes
+ an issue with converting databases. Thanks to Kevin Smith for
+ discovering this.
+
+- Added logging to the pack code to help diagnose performance issues.
+
+- Additions to the object_ref table are now periodically committed
+ during pre_pack so that the work is not lost if pre_pack fails.
+
+- Modified the pack code to pack one transaction at a time and
+ release the commit lock frequently. This should help large pack
+ operations.
+
+- Fixed buildout-based installation of the zodbconvert script. Thanks to
+ Jim Fulton.
+
+
+RelStorage 1.0.1
+
+- The speedtest script failed if run on a test database that has no tables.
+ Now the script creates the tables if needed. Thanks to Flavio Coelho
+ for discovering this.
+
+- Reworked the auto-reconnect logic so that applications never see
+ temporary database disconnects if possible. Thanks to Rigel Di Scala
+ for pointing out this issue.
+
+- Improved the log messages explaining database connection failures.
+
+- Moved poll_invalidations to the common adapter base class, reducing the
+ amount of code to maintain.
+
+
+RelStorage 1.0
+
+- Added a utility for converting between storages called zodbconvert.
+
+
+RelStorage 1.0c1
+
+- The previous fix for non-ASCII characters was incorrect. Now transaction
+ metadata is stored as raw bytes. A schema migration is required; see
+ notes/migrate-1.0-beta.txt.
+
+- Integrated setuptools and made an egg.
+
+
+RelStorage 1.0 beta
+
+- Renamed to reflect expanding database support.
+
+- Added support for Oracle 10g.
+
+- Major overhaul with many scalability and reliability improvements,
+ particularly in the area of packing.
+
+- Moved to svn.zope.org and switched to ZPL 2.1.
+
+- Made two-phase commit optional in both Oracle and PostgreSQL. They
+ both use commit_lock in such a way that the commit is not likely to
+ fail in the second phase.
+
+- Switched most database transaction isolation levels from serializable
+ to read committed. It turns out that commit_lock already provides
+ the serializability guarantees we need, so it is safe to take advantage
+ of the potential speed gains. The one major exception is the load
+ connection, which requires an unchanging view of the database.
+
+- Stored objects are now buffered in a database table rather than a file.
+
+- Stopped using the LISTEN and NOTIFY statements in PostgreSQL since
+ they are not strictly transactional in the sense we require.
+
+- Started using a prepared statement in PostgreSQL for getting the
+ newest transaction ID quickly.
+
+- Removed the code in the Oracle adapter for retrying connection attempts.
+ (It is better to just reconfigure Oracle.)
+
+- Added support for MySQL 5.0.
+
+- Added the poll_interval option. It reduces the frequency of database
+ polls, but it also increases the potential for conflict errors on
+ servers with high write volume.
+
+- Implemented the storage iterator protocol, making it possible to copy
+ transactions to and from FileStorage and other RelStorage instances.
+
+- Fixed a bug that caused OIDs to be reused after importing transactions.
+ Added a corresponding test.
+
+- Made it possible to disable garbage collection during packing.
+ Exposed the option in zope.conf.
+
+- Valery Suhomlinov discovered a problem with non-ASCII data in transaction
+ metadata. The problem has been fixed for all supported databases.
+
+
+PGStorage 0.4
+
+- Began using the PostgreSQL LISTEN and NOTIFY statements as a shortcut
+ for invalidation polling.
+
+- Removed the commit_order code. The commit_order idea was intended to
+ allow concurrent commits, but that idea is a little too ambitious while
+ other more important ideas are being tested. Something like it may
+ come later.
+
+- Improved connection management: only one database connection is
+ held continuously open per storage instance.
+
+- Reconnect to the database automatically.
+
+- Removed test mode.
+
+- Switched from using a ZODB.Connection subclass to a ZODB patch. The
+ Connection class changes in subtle ways too often to subclass reliably;
+ a patch is much safer.
+
+- PostgreSQL 8.1 is now a dependency because PGStorage uses two phase commit.
+
+- Fixed an undo bug. Symptom: attempting to examine the undo log revealed
+ broken pickles. Cause: the extension field was not being wrapped in
+ psycopg2.Binary upon insert. Solution: used psycopg2.Binary.
+ Unfortunately, this doesn't fix existing transactions people have
+ committed. If anyone has any data to keep, fixing the old transactions
+ should be easy.
+
+- Moved from a private CVS repository to Sourceforge.
+ See http://pgstorage.sourceforge.net . Also switched to the MIT license.
+
+- David Pratt added a basic getSize() implementation so that the Zope
+ management interface displays an estimate of the size of the database.
+
+- Turned PGStorage into a top-level package. Python generally makes
+ top-level packages easier to install.
+
+
+PGStorage 0.3
+
+- Made compatible with Zope 3, although an undo bug apparently remains.
+
+
+PGStorage 0.2
+
+- Fixed concurrent commits, which were generating deadlocks. Fixed by
+ adding a special table, "commit_lock", which is used for
+ synchronizing increments of commit_seq (but only at final commit.)
+ If you are upgrading from version 0.1, you need to change your
+ database using the 'psql' prompt:
+
+ create table commit_lock ();
+
+- Added speed tests and an OpenDocument spreadsheet comparing
+ FileStorage / ZEO with PGStorage. PGStorage wins at reading objects
+ and writing a lot of small transactions, while FileStorage / ZEO
+ wins at writing big transactions. Interestingly, they tie when
+ writing a RAM disk.
+
Deleted: relstorage/tags/1.1c1/README.txt
===================================================================
--- relstorage/branches/1.1/README.txt 2008-07-23 18:06:37 UTC (rev 88770)
+++ relstorage/tags/1.1c1/README.txt 2008-08-03 00:12:31 UTC (rev 89242)
@@ -1,232 +0,0 @@
-
-
-Overview
-========
-
- RelStorage is a storage implementation for ZODB that stores pickles in a
-relational database. PostgreSQL 8.1 and above (via psycopg2), MySQL 5.0.x (via
-MySQLdb), and Oracle 10g (via cx_Oracle) are currently supported.
-
- RelStorage replaces the PGStorage project.
-
- See:
-
- http://wiki.zope.org/ZODB/RelStorage (wiki)
- http://shane.willowrise.com/ (blog)
- http://pypi.python.org/pypi/RelStorage (PyPI entry and downloads)
-
-
-Features
-========
-
- * It is a drop-in replacement for FileStorage and ZEO.
- * There is a simple way to convert FileStorage to RelStorage and back again.
-You can also convert a RelStorage instance to a different relational database.
- * Designed for high volume sites: multiple ZODB instances can share the same
-database. This is similar to ZEO, but RelStorage does not require ZEO.
- * According to some tests, RelStorage handles high concurrency better than
-the standard combination of ZEO and FileStorage.
- * Whereas FileStorage takes longer to start as the database grows due to an
-in-memory index of all objects, RelStorage starts quickly regardless of
-database size.
- * Supports undo and packing.
- * Free, open source (ZPL 2.1)
-
-
-Installation in Zope
-====================
-
- You can install RelStorage using easy_install::
-
- easy_install RelStorage
-
- If you are not using easy_install (part of the setuptools package), you can
-get the latest release at PyPI (http://pypi.python.org/pypi/RelStorage), then
-place the relstorage package in the lib/python directory of either the
-SOFTWARE_HOME or the INSTANCE_HOME. You can do this with the following
-command::
-
- python2.4 setup.py install --install-lib=${INSTANCE_HOME}/lib/python
-
- Before you can use RelStorage, ZODB must have the invalidation polling patch
-applied. Get it from Subversion (http://svn.zope.org/relstorage/trunk/).
-There are two versions of the patch: one for ZODB 3.7.1 (which is part of Zope
-2.10.5) and one for ZODB 3.8.0 (which is part of Zope 2.11). The patch has no
-effect on ZODB except when using RelStorage. Hopefully, a future release of
-ZODB will include the feature.
-
- You need the Python database adapter that corresponds with your database.
-Install psycopg2, MySQLdb 1.2.2+, or cx_Oracle 4.3+. Note that Debian Etch
-ships MySQLdb 1.2.1, but that version has a bug in BLOB handling that manifests
-itself only with certain character set configurations. MySQLdb 1.2.2 fixes the
-bug. Also, MySQL 5.1.23 has major bugs that lead to loss of data when packing,
-so MySQL 5.1 is not recommended at this time.
-
- Finally, modify etc/zope.conf of your Zope instance. Remove the main mount
-point and add one of the following blocks. For PostgreSQL::
-
- %import relstorage
- <zodb_db main>
- mount-point /
- <relstorage>
- <postgresql>
- # The dsn is optional, as are each of the parameters in the dsn.
- dsn dbname='zodb' user='username' host='localhost' password='pass'
- </postgresql>
- </relstorage>
- </zodb_db>
-
- For MySQL::
-
- %import relstorage
- <zodb_db main>
- mount-point /
- <relstorage>
- <mysql>
- # Most of the options provided by MySQLdb are available.
- # See component.xml.
- db zodb
- </mysql>
- </relstorage>
- </zodb_db>
-
- For Oracle (10g XE in this example)::
-
- %import relstorage
- <zodb_db main>
- mount-point /
- <relstorage>
- <oracle>
- user username
- password pass
- dsn XE
- </oracle>
- </relstorage>
- </zodb_db>
-
-Migrating from FileStorage
-==========================
-
- You can convert a FileStorage instance to RelStorage and back using a utility
-called ZODBConvert. See http://wiki.zope.org/ZODB/ZODBConvert .
-
-
-Migrating from PGStorage
-========================
-
- The following script migrates your database from PGStorage to RelStorage 1.0
-beta:
-
- migrate.sql_
-
- .. _migrate.sql:
-http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate.sql
-
- After you do this, you still need to migrate from 1.0 beta to the latest
-release.
-
-
-Migrating to a new version of RelStorage
-========================================
-
- Sometimes RelStorage needs a schema modification along with a software
-upgrade. Hopefully, this will not often be necessary.
-
- To migrate from version 1.0 beta to version 1.0c1, see:
-
- migrate-1.0-beta.txt_
-
- .. _migrate-1.0-beta.txt:
-http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate-1.0-beta.txt
-
- To migrate from version 1.0.1 to version 1.1b1, see:
-
- migrate-1.0.1.txt_
-
- .. _migrate-1.0.1.txt:
-http://svn.zope.org/*checkout*/relstorage/branches/1.1/notes/migrate-1.0.1.txt
-
-
-Optional Features
-=================
-
- poll-interval
-
- This option is useful if you need to reduce database traffic. If set,
-RelStorage will poll the database for changes less often. A setting of 1 to 5
-seconds should be sufficient for most systems. Fractional seconds are allowed.
-
- While this setting should not affect database integrity, it increases the
-probability of basing transactions on stale data, leading to conflicts. Thus a
-nonzero setting can hurt the performance of servers with high write volume.
-
- To enable this feature, add a line similar to "poll-interval 2" inside a
-<relstorage> section of zope.conf.
-
- pack-gc
-
- If pack-gc is false, pack operations do not perform garbage collection.
-Garbage collection is enabled by default.
-
- If garbage collection is disabled, pack operations keep at least one
-revision of every object. With garbage collection disabled, the pack code does
-not need to follow object references, making packing conceivably much faster.
-However, some of that benefit may be lost due to an ever increasing number of
-unused objects.
-
- Disabling garbage collection is also a hack that ensures inter-database
-references never break.
-
- To disable garbage collection, add the line "pack-gc no" inside a
-<relstorage> section of zope.conf.
-
-
-Development
-===========
-
- You can checkout from Subversion using the following command::
-
- svn co svn://svn.zope.org/repos/main/relstorage/trunk RelStorage
-
- You can also browse the code:
-
- http://svn.zope.org/relstorage/trunk/
-
- The best place to discuss development of RelStorage is on the zodb-dev
-mailing list.
-
-
-
-Probable FAQs
-==============
-
- Q: How can I help?
-
- A: The best way to help is to test and to provide database-specific
-expertise. Ask questions about RelStorage on the zodb-dev mailing list.
-
- Q: Can I perform SQL queries on the data in the database?
-
- A: No. Like FileStorage and DirectoryStorage, RelStorage stores the data
-as pickles, making it hard for anything but ZODB to interpret the data. An
-earlier project called Ape attempted to store data in a truly relational way,
-but it turned out that Ape worked too much against ZODB principles and
-therefore could not be made reliable enough for production use. RelStorage, on
-the other hand, is much closer to an ordinary ZODB storage, and is therefore
-much safer for production use.
-
- Q: How does RelStorage performance compare with FileStorage?
-
- A: According to benchmarks, RelStorage with PostgreSQL is often faster than
-FileStorage, especially under high concurrency.
-
- Q: Why should I choose RelStorage?
-
- A: Because RelStorage is a fairly small layer that builds on world-class
-databases. These databases have proven reliability and scalability, along with
-numerous support options.
-
- Q: Can RelStorage replace ZRS (Zope Replication Services)?
-
- A: In theory, yes. With RelStorage, you can use the replication features
-native to your database. However, this capability has not yet been tested.
Copied: relstorage/tags/1.1c1/README.txt (from rev 89241, relstorage/branches/1.1/README.txt)
===================================================================
--- relstorage/tags/1.1c1/README.txt (rev 0)
+++ relstorage/tags/1.1c1/README.txt 2008-08-03 00:12:31 UTC (rev 89242)
@@ -0,0 +1,280 @@
+
+
+Overview
+========
+
+ RelStorage is a storage implementation for ZODB that stores pickles in a
+relational database. PostgreSQL 8.1 and above (via psycopg2), MySQL 5.0.x (via
+MySQLdb), and Oracle 10g (via cx_Oracle) are currently supported.
+
+ RelStorage replaces the PGStorage project.
+
+ See:
+
+ http://wiki.zope.org/ZODB/RelStorage (wiki)
+ http://shane.willowrise.com/ (blog)
+ http://pypi.python.org/pypi/RelStorage (PyPI entry and downloads)
+
+
+Features
+========
+
+ * It is a drop-in replacement for FileStorage and ZEO.
+ * There is a simple way to convert FileStorage to RelStorage and back again.
+You can also convert a RelStorage instance to a different relational database.
+ * Designed for high volume sites: multiple ZODB instances can share the same
+database. This is similar to ZEO, but RelStorage does not require ZEO.
+ * According to some tests, RelStorage handles high concurrency better than
+the standard combination of ZEO and FileStorage.
+ * Whereas FileStorage takes longer to start as the database grows due to an
+in-memory index of all objects, RelStorage starts quickly regardless of
+database size.
+ * Supports undo and packing.
+ * Free, open source (ZPL 2.1)
+
+
+Installation in Zope
+====================
+
+ You can install RelStorage using easy_install::
+
+ easy_install RelStorage
+
+ If you are not using easy_install (part of the setuptools package), you can
+get the latest release at PyPI (http://pypi.python.org/pypi/RelStorage), then
+place the relstorage package in the lib/python directory of either the
+SOFTWARE_HOME or the INSTANCE_HOME. You can do this with the following
+command::
+
+ python2.4 setup.py install --install-lib=${INSTANCE_HOME}/lib/python
+
+ Before you can use RelStorage, ZODB must have the invalidation polling patch
+applied. Get it from Subversion (http://svn.zope.org/relstorage/trunk/).
+There are two versions of the patch: one for ZODB 3.7.1 (which is part of Zope
+2.10.5) and one for ZODB 3.8.0 (which is part of Zope 2.11). The patch has no
+effect on ZODB except when using RelStorage. Hopefully, a future release of
+ZODB will include the feature.
+
+ You need the Python database adapter that corresponds with your database.
+Install psycopg2, MySQLdb 1.2.2+, or cx_Oracle 4.3+. Note that Debian Etch
+ships MySQLdb 1.2.1, but that version has a bug in BLOB handling that manifests
+itself only with certain character set configurations. MySQLdb 1.2.2 fixes the
+bug. Also, MySQL 5.1.23 has major bugs that lead to loss of data when packing,
+so MySQL 5.1 is not recommended at this time.
+
+ Finally, modify etc/zope.conf of your Zope instance. Remove the main mount
+point and add one of the following blocks. For PostgreSQL::
+
+ %import relstorage
+ <zodb_db main>
+ mount-point /
+ <relstorage>
+ <postgresql>
+ # The dsn is optional, as are each of the parameters in the dsn.
+ dsn dbname='zodb' user='username' host='localhost' password='pass'
+ </postgresql>
+ </relstorage>
+ </zodb_db>
+
+ For MySQL::
+
+ %import relstorage
+ <zodb_db main>
+ mount-point /
+ <relstorage>
+ <mysql>
+ # Most of the options provided by MySQLdb are available.
+ # See component.xml.
+ db zodb
+ </mysql>
+ </relstorage>
+ </zodb_db>
+
+ For Oracle (10g XE in this example)::
+
+ %import relstorage
+ <zodb_db main>
+ mount-point /
+ <relstorage>
+ <oracle>
+ user username
+ password pass
+ dsn XE
+ </oracle>
+ </relstorage>
+ </zodb_db>
+
+Migrating from FileStorage
+==========================
+
+ You can convert a FileStorage instance to RelStorage and back using a utility
+called ZODBConvert. See http://wiki.zope.org/ZODB/ZODBConvert .
+
+
+Migrating from PGStorage
+========================
+
+ The following script migrates your database from PGStorage to RelStorage 1.0
+beta:
+
+ migrate.sql_
+
+ .. _migrate.sql:
+http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate.sql
+
+ After you do this, you still need to migrate from 1.0 beta to the latest
+release.
+
+
+Migrating to a new version of RelStorage
+========================================
+
+ Sometimes RelStorage needs a schema modification along with a software
+upgrade. Hopefully, this will not often be necessary.
+
+ To migrate from version 1.0 beta to version 1.0c1, see:
+
+ migrate-1.0-beta.txt_
+
+ .. _migrate-1.0-beta.txt:
+http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate-1.0-beta.txt
+
+ To migrate from version 1.0.1 to version 1.1, see:
+
+ migrate-1.0.1.txt_
+
+ .. _migrate-1.0.1.txt:
+http://svn.zope.org/*checkout*/relstorage/branches/1.1/notes/migrate-1.0.1.txt
+
+
+Optional Features
+=================
+
+ Specify these options in zope.conf.
+
+ poll-interval
+
+ Defer polling the database for the specified maximum time interval.
+ Set to 0 (the default) to always poll. Fractional seconds are
+ allowed.
+
+ Use this to lighten the database load on servers with high read
+ volume and low write volume. A setting of 1-5 seconds is sufficient
+ for most systems.
+
+ While this setting should not affect database integrity,
+ it increases the probability of basing transactions on stale data,
+ leading to conflicts. Thus a nonzero setting can hurt
+ the performance of servers with high write volume.
+
+ pack-gc
+
+ If pack-gc is false, pack operations do not perform
+ garbage collection. Garbage collection is enabled by default.
+
+ If garbage collection is disabled, pack operations keep at least one
+ revision of every object. With garbage collection disabled, the
+ pack code does not need to follow object references, making
+ packing conceivably much faster. However, some of that benefit
+ may be lost due to an ever increasing number of unused objects.
+
+ Disabling garbage collection is also a hack that ensures
+ inter-database references never break.
+
+ pack-batch-timeout
+
+ Packing occurs in batches of transactions; this specifies the
+ timeout in seconds for each batch. Note that some database
+ configurations have unpredictable I/O performance
+ and might stall much longer than the timeout.
+ The default timeout is 5.0 seconds.
+
+ pack-duty-cycle
+
+ After each batch, the pack code pauses for a time to
+ allow concurrent transactions to commit. The pack-duty-cycle
+ specifies what fraction of time should be spent on packing.
+ For example, if the duty cycle is 0.75, then 75% of the time
+ will be spent packing: a 6 second pack batch
+ will be followed by a 2 second delay. The duty cycle should
+ be greater than 0.0 and less than or equal to 1.0. Specify
+ 1.0 for no delay between batches.
+
+ The default is 0.5. Raise it to finish packing faster; lower it
+ to reduce the effect of packing on transaction commit performance.
+
+ pack-max-delay
+
+ This specifies a maximum delay between pack batches. Sometimes
+ the database takes an extra long time to finish a pack batch; at
+ those times it is useful to cap the delay imposed by the
+ pack-duty-cycle. The default is 20 seconds.
+
+ cache-servers
+
+ Specifies a list of memcache servers. Enabling memcache integration
+ is useful if the connection to the relational database has high
+ latency and the connection to memcache has significantly lower
+ latency. On the other hand, if the connection to the relational
+ database already has low latency, memcache integration may actually
+ hurt overall performance.
+
+ Provide a list of host:port pairs, separated by whitespace.
+ "127.0.0.1:11211" is a common setting. The default is to disable
+ memcache integration.
+
+ cache-module-name
+
+ Specifies which Python memcache module to use. The default is
+ "memcache", a pure Python module. An alternative module is
+ "cmemcache". This setting has no effect unless cache-servers is set.
+
+Development
+===========
+
+ You can checkout from Subversion using the following command::
+
+ svn co svn://svn.zope.org/repos/main/relstorage/trunk RelStorage
+
+ You can also browse the code:
+
+ http://svn.zope.org/relstorage/trunk/
+
+ The best place to discuss development of RelStorage is on the zodb-dev
+mailing list.
+
+
+
+Probable FAQs
+==============
+
+ Q: How can I help?
+
+ A: The best way to help is to test and to provide database-specific
+expertise. Ask questions about RelStorage on the zodb-dev mailing list.
+
+ Q: Can I perform SQL queries on the data in the database?
+
+ A: No. Like FileStorage and DirectoryStorage, RelStorage stores the data
+as pickles, making it hard for anything but ZODB to interpret the data. An
+earlier project called Ape attempted to store data in a truly relational way,
+but it turned out that Ape worked too much against ZODB principles and
+therefore could not be made reliable enough for production use. RelStorage, on
+the other hand, is much closer to an ordinary ZODB storage, and is therefore
+much safer for production use.
+
+ Q: How does RelStorage performance compare with FileStorage?
+
+ A: According to benchmarks, RelStorage with PostgreSQL is often faster than
+FileStorage, especially under high concurrency.
+
+ Q: Why should I choose RelStorage?
+
+ A: Because RelStorage is a fairly small layer that builds on world-class
+databases. These databases have proven reliability and scalability, along with
+numerous support options.
+
+ Q: Can RelStorage replace ZRS (Zope Replication Services)?
+
+ A: In theory, yes. With RelStorage, you can use the replication features
+native to your database. However, this capability has not yet been tested.
Deleted: relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-7-1.patch
===================================================================
--- relstorage/branches/1.1/poll-invalidation-1-zodb-3-7-1.patch 2008-07-23 18:06:37 UTC (rev 88770)
+++ relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-7-1.patch 2008-08-03 00:12:31 UTC (rev 89242)
@@ -1,94 +0,0 @@
-diff -r 34747fbd09ec Connection.py
---- a/Connection.py Tue Nov 20 21:57:31 2007 -0700
-+++ b/Connection.py Fri Jan 11 21:19:00 2008 -0700
-@@ -75,8 +75,14 @@ class Connection(ExportImport, object):
- """Create a new Connection."""
-
- self._db = db
-- self._normal_storage = self._storage = db._storage
-- self.new_oid = db._storage.new_oid
-+ storage = db._storage
-+ m = getattr(storage, 'bind_connection', None)
-+ if m is not None:
-+ # Use a storage instance bound to this connection.
-+ storage = m(self)
-+
-+ self._normal_storage = self._storage = storage
-+ self.new_oid = storage.new_oid
- self._savepoint_storage = None
-
- self.transaction_manager = self._synch = self._mvcc = None
-@@ -170,6 +176,12 @@ class Connection(ExportImport, object):
- # Multi-database support
- self.connections = {self._db.database_name: self}
-
-+ # Allow the storage to decide whether invalidations should
-+ # propagate between connections. If the storage provides MVCC
-+ # semantics, it is better to not propagate invalidations between
-+ # connections.
-+ self._propagate_invalidations = getattr(
-+ self._storage, 'propagate_invalidations', True)
-
- def add(self, obj):
- """Add a new object 'obj' to the database and assign it an oid."""
-@@ -267,6 +279,11 @@ class Connection(ExportImport, object):
- self.transaction_manager.unregisterSynch(self)
- self._synch = None
-
-+ # If the storage wants to know, tell it this connection is closing.
-+ m = getattr(self._storage, 'connection_closing', None)
-+ if m is not None:
-+ m()
-+
- if primary:
- for connection in self.connections.values():
- if connection is not self:
-@@ -295,6 +312,10 @@ class Connection(ExportImport, object):
-
- def invalidate(self, tid, oids):
- """Notify the Connection that transaction 'tid' invalidated oids."""
-+ if not self._propagate_invalidations:
-+ # The storage disabled inter-connection invalidation.
-+ return
-+
- self._inv_lock.acquire()
- try:
- if self._txn_time is None:
-@@ -438,8 +459,23 @@ class Connection(ExportImport, object):
- self._registered_objects = []
- self._creating.clear()
-
-+ def _poll_invalidations(self):
-+ """Poll and process object invalidations provided by the storage.
-+ """
-+ m = getattr(self._storage, 'poll_invalidations', None)
-+ if m is not None:
-+ # Poll the storage for invalidations.
-+ invalidated = m()
-+ if invalidated is None:
-+ # special value: the transaction is so old that
-+ # we need to flush the whole cache.
-+ self._cache.invalidate(self._cache.cache_data.keys())
-+ elif invalidated:
-+ self._cache.invalidate(invalidated)
-+
- # Process pending invalidations.
- def _flush_invalidations(self):
-+ self._poll_invalidations()
- self._inv_lock.acquire()
- try:
- # Non-ghostifiable objects may need to read when they are
-diff -r 34747fbd09ec DB.py
---- a/DB.py Tue Nov 20 21:57:31 2007 -0700
-+++ b/DB.py Wed Nov 28 18:33:12 2007 -0700
-@@ -260,6 +260,10 @@ class DB(object):
- storage.store(z64, None, file.getvalue(), '', t)
- storage.tpc_vote(t)
- storage.tpc_finish(t)
-+ if hasattr(storage, 'connection_closing'):
-+ # Let the storage release whatever resources it used for loading
-+ # the root object.
-+ storage.connection_closing()
-
- # Multi-database setup.
- if databases is None:
Copied: relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-7-1.patch (from rev 89241, relstorage/branches/1.1/poll-invalidation-1-zodb-3-7-1.patch)
===================================================================
--- relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-7-1.patch (rev 0)
+++ relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-7-1.patch 2008-08-03 00:12:31 UTC (rev 89242)
@@ -0,0 +1,96 @@
+Index: Connection.py
+===================================================================
+--- Connection.py (revision 87280)
++++ Connection.py (working copy)
+@@ -75,8 +75,14 @@
+ """Create a new Connection."""
+
+ self._db = db
+- self._normal_storage = self._storage = db._storage
+- self.new_oid = db._storage.new_oid
++ storage = db._storage
++ m = getattr(storage, 'bind_connection', None)
++ if m is not None:
++ # Use a storage instance bound to this connection.
++ storage = m(self)
++
++ self._normal_storage = self._storage = storage
++ self.new_oid = storage.new_oid
+ self._savepoint_storage = None
+
+ self.transaction_manager = self._synch = self._mvcc = None
+@@ -170,6 +176,12 @@
+ # Multi-database support
+ self.connections = {self._db.database_name: self}
+
++ # Allow the storage to decide whether invalidations should
++ # propagate between connections. If the storage provides MVCC
++ # semantics, it is better to not propagate invalidations between
++ # connections.
++ self._propagate_invalidations = getattr(
++ self._storage, 'propagate_invalidations', True)
+
+ def add(self, obj):
+ """Add a new object 'obj' to the database and assign it an oid."""
+@@ -267,6 +279,11 @@
+ self.transaction_manager.unregisterSynch(self)
+ self._synch = None
+
++ # If the storage wants to know, tell it this connection is closing.
++ m = getattr(self._storage, 'connection_closing', None)
++ if m is not None:
++ m()
++
+ if primary:
+ for connection in self.connections.values():
+ if connection is not self:
+@@ -295,6 +312,10 @@
+
+ def invalidate(self, tid, oids):
+ """Notify the Connection that transaction 'tid' invalidated oids."""
++ if not self._propagate_invalidations:
++ # The storage disabled inter-connection invalidation.
++ return
++
+ self._inv_lock.acquire()
+ try:
+ if self._txn_time is None:
+@@ -438,8 +459,23 @@
+ self._registered_objects = []
+ self._creating.clear()
+
++ def _poll_invalidations(self):
++ """Poll and process object invalidations provided by the storage.
++ """
++ m = getattr(self._storage, 'poll_invalidations', None)
++ if m is not None:
++ # Poll the storage for invalidations.
++ invalidated = m()
++ if invalidated is None:
++ # special value: the transaction is so old that
++ # we need to flush the whole cache.
++ self._cache.invalidate(self._cache.cache_data.keys())
++ elif invalidated:
++ self._cache.invalidate(invalidated)
++
+ # Process pending invalidations.
+ def _flush_invalidations(self):
++ self._poll_invalidations()
+ self._inv_lock.acquire()
+ try:
+ # Non-ghostifiable objects may need to read when they are
+Index: DB.py
+===================================================================
+--- DB.py (revision 87280)
++++ DB.py (working copy)
+@@ -260,6 +260,10 @@
+ storage.store(z64, None, file.getvalue(), '', t)
+ storage.tpc_vote(t)
+ storage.tpc_finish(t)
++ if hasattr(storage, 'connection_closing'):
++ # Let the storage release whatever resources it used for loading
++ # the root object.
++ storage.connection_closing()
+
+ # Multi-database setup.
+ if databases is None:
Deleted: relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch
===================================================================
--- relstorage/branches/1.1/poll-invalidation-1-zodb-3-8-0.patch 2008-07-23 18:06:37 UTC (rev 88770)
+++ relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch 2008-08-03 00:12:31 UTC (rev 89242)
@@ -1,95 +0,0 @@
-diff -r 72bf2fd94b66 src/ZODB/Connection.py
---- a/src/ZODB/Connection.py Wed Jan 30 23:23:05 2008 -0700
-+++ b/src/ZODB/Connection.py Wed Jan 30 23:38:51 2008 -0700
-@@ -90,8 +90,15 @@ class Connection(ExportImport, object):
- self.connections = {self._db.database_name: self}
-
- self._version = version
-- self._normal_storage = self._storage = db._storage
-- self.new_oid = db._storage.new_oid
-+
-+ storage = db._storage
-+ m = getattr(storage, 'bind_connection', None)
-+ if m is not None:
-+ # Use a storage instance bound to this connection.
-+ storage = m(self)
-+ self._normal_storage = self._storage = storage
-+ self.new_oid = storage.new_oid
-+
- self._savepoint_storage = None
-
- # Do we need to join a txn manager?
-@@ -151,6 +158,12 @@ class Connection(ExportImport, object):
- # in the cache on abort and in other connections on finish.
- self._modified = []
-
-+ # Allow the storage to decide whether invalidations should
-+ # propagate between connections. If the storage provides MVCC
-+ # semantics, it is better to not propagate invalidations between
-+ # connections.
-+ self._propagate_invalidations = getattr(
-+ self._storage, 'propagate_invalidations', True)
-
- # _invalidated queues invalidate messages delivered from the DB
- # _inv_lock prevents one thread from modifying the set while
-@@ -297,6 +310,11 @@ class Connection(ExportImport, object):
- if self._opened:
- self.transaction_manager.unregisterSynch(self)
-
-+ # If the storage wants to know, tell it this connection is closing.
-+ m = getattr(self._storage, 'connection_closing', None)
-+ if m is not None:
-+ m()
-+
- if primary:
- for connection in self.connections.values():
- if connection is not self:
-@@ -328,6 +346,10 @@ class Connection(ExportImport, object):
-
- def invalidate(self, tid, oids):
- """Notify the Connection that transaction 'tid' invalidated oids."""
-+ if not self._propagate_invalidations:
-+ # The storage disabled inter-connection invalidation.
-+ return
-+
- self._inv_lock.acquire()
- try:
- if self._txn_time is None:
-@@ -469,8 +491,23 @@ class Connection(ExportImport, object):
- self._registered_objects = []
- self._creating.clear()
-
-+ def _poll_invalidations(self):
-+ """Poll and process object invalidations provided by the storage.
-+ """
-+ m = getattr(self._storage, 'poll_invalidations', None)
-+ if m is not None:
-+ # Poll the storage for invalidations.
-+ invalidated = m()
-+ if invalidated is None:
-+ # special value: the transaction is so old that
-+ # we need to flush the whole cache.
-+ self._cache.invalidate(self._cache.cache_data.keys())
-+ elif invalidated:
-+ self._cache.invalidate(invalidated)
-+
- # Process pending invalidations.
- def _flush_invalidations(self):
-+ self._poll_invalidations()
- self._inv_lock.acquire()
- try:
- # Non-ghostifiable objects may need to read when they are
-diff -r 72bf2fd94b66 src/ZODB/DB.py
---- a/src/ZODB/DB.py Wed Jan 30 23:23:05 2008 -0700
-+++ b/src/ZODB/DB.py Wed Jan 30 23:38:51 2008 -0700
-@@ -284,6 +284,10 @@ class DB(object):
- storage.store(z64, None, file.getvalue(), '', t)
- storage.tpc_vote(t)
- storage.tpc_finish(t)
-+ if hasattr(storage, 'connection_closing'):
-+ # Let the storage release whatever resources it used for loading
-+ # the root object.
-+ storage.connection_closing()
-
- # Multi-database setup.
- if databases is None:
Copied: relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch (from rev 89241, relstorage/branches/1.1/poll-invalidation-1-zodb-3-8-0.patch)
===================================================================
--- relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch (rev 0)
+++ relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch 2008-08-03 00:12:31 UTC (rev 89242)
@@ -0,0 +1,97 @@
+Index: Connection.py
+===================================================================
+--- Connection.py (revision 87666)
++++ Connection.py (working copy)
+@@ -90,8 +90,15 @@
+ self.connections = {self._db.database_name: self}
+
+ self._version = version
+- self._normal_storage = self._storage = db._storage
+- self.new_oid = db._storage.new_oid
++
++ storage = db._storage
++ m = getattr(storage, 'bind_connection', None)
++ if m is not None:
++ # Use a storage instance bound to this connection.
++ storage = m(self)
++ self._normal_storage = self._storage = storage
++ self.new_oid = storage.new_oid
++
+ self._savepoint_storage = None
+
+ # Do we need to join a txn manager?
+@@ -151,6 +158,12 @@
+ # in the cache on abort and in other connections on finish.
+ self._modified = []
+
++ # Allow the storage to decide whether invalidations should
++ # propagate between connections. If the storage provides MVCC
++ # semantics, it is better to not propagate invalidations between
++ # connections.
++ self._propagate_invalidations = getattr(
++ self._storage, 'propagate_invalidations', True)
+
+ # _invalidated queues invalidate messages delivered from the DB
+ # _inv_lock prevents one thread from modifying the set while
+@@ -297,6 +310,11 @@
+ if self._opened:
+ self.transaction_manager.unregisterSynch(self)
+
++ # If the storage wants to know, tell it this connection is closing.
++ m = getattr(self._storage, 'connection_closing', None)
++ if m is not None:
++ m()
++
+ if primary:
+ for connection in self.connections.values():
+ if connection is not self:
+@@ -328,6 +346,10 @@
+
+ def invalidate(self, tid, oids):
+ """Notify the Connection that transaction 'tid' invalidated oids."""
++ if not self._propagate_invalidations:
++ # The storage disabled inter-connection invalidation.
++ return
++
+ self._inv_lock.acquire()
+ try:
+ if self._txn_time is None:
+@@ -469,8 +491,23 @@
+ self._registered_objects = []
+ self._creating.clear()
+
++ def _poll_invalidations(self):
++ """Poll and process object invalidations provided by the storage.
++ """
++ m = getattr(self._storage, 'poll_invalidations', None)
++ if m is not None:
++ # Poll the storage for invalidations.
++ invalidated = m()
++ if invalidated is None:
++ # special value: the transaction is so old that
++ # we need to flush the whole cache.
++ self._cache.invalidate(self._cache.cache_data.keys())
++ elif invalidated:
++ self._cache.invalidate(invalidated)
++
+ # Process pending invalidations.
+ def _flush_invalidations(self):
++ self._poll_invalidations()
+ self._inv_lock.acquire()
+ try:
+ # Non-ghostifiable objects may need to read when they are
+Index: DB.py
+===================================================================
+--- DB.py (revision 87666)
++++ DB.py (working copy)
+@@ -284,6 +284,10 @@
+ storage.store(z64, None, file.getvalue(), '', t)
+ storage.tpc_vote(t)
+ storage.tpc_finish(t)
++ if hasattr(storage, 'connection_closing'):
++ # Let the storage release whatever resources it used for loading
++ # the root object.
++ storage.connection_closing()
+
+ # Multi-database setup.
+ if databases is None:
Copied: relstorage/tags/1.1c1/relstorage (from rev 89241, relstorage/branches/1.1/relstorage)
More information about the Checkins
mailing list