[Checkins] SVN: relstorage/branches/1.1/ Merged from trunk (-r95060:95068)
Shane Hathaway
shane at hathawaymix.org
Tue Jan 27 00:56:48 EST 2009
Log message for revision 95070:
Merged from trunk (-r95060:95068)
Changed:
U relstorage/branches/1.1/CHANGES.txt
U relstorage/branches/1.1/README.txt
U relstorage/branches/1.1/setup.py
-=-
Modified: relstorage/branches/1.1/CHANGES.txt
===================================================================
--- relstorage/branches/1.1/CHANGES.txt 2009-01-27 05:46:41 UTC (rev 95069)
+++ relstorage/branches/1.1/CHANGES.txt 2009-01-27 05:56:48 UTC (rev 95070)
@@ -1,4 +1,6 @@
+
Next Release
+------------
- When both cache-servers and poll-interval are set, we now poll the
cache for changes on every request. This makes it possible to use
@@ -22,6 +24,7 @@
Version 1.1.1 (2008-12-27)
+--------------------------
- Worked around MySQL performance bugs in packing. Used temporary
tables and another column in the pack_object table. The
@@ -32,6 +35,7 @@
Version 1.1 (2008-12-19)
+------------------------
- Normalized poll-invalidation patches as Solaris' patch command would not
accept the current format. The patches now apply with:
@@ -39,6 +43,7 @@
- In MySQL, Use DROP TABLE IF EXISTS instead of TRUNCATE to clear 'temp_store'
because:
+
- TRUNCATE has one page of caveats in the MySQL documentation.
- TEMPORARY TABLEs have half a page of caveats when it comes to
replication.
@@ -62,6 +67,7 @@
Version 1.1c1
+-------------
- Added optional memcache integration. This is useful when the connection
to the relational database has high latency.
@@ -72,6 +78,7 @@
Version 1.1b2
+-------------
- Made the MySQL locks database-specific rather than server-wide. This is
important for multi-database configurations.
@@ -85,6 +92,7 @@
Version 1.1b1
+-------------
- Fixed the use of setup.py without setuptools. Thanks to Chris Withers.
@@ -106,6 +114,7 @@
Version 1.0.1
+-------------
- The speedtest script failed if run on a test database that has no tables.
Now the script creates the tables if needed. Thanks to Flavio Coelho
@@ -122,11 +131,13 @@
Version 1.0
+-----------
- Added a utility for converting between storages called zodbconvert.
Version 1.0c1
+-------------
- The previous fix for non-ASCII characters was incorrect. Now transaction
metadata is stored as raw bytes. A schema migration is required; see
@@ -136,6 +147,7 @@
Version 1.0 beta
+----------------
- Renamed to reflect expanding database support.
@@ -190,6 +202,7 @@
=================
Version 0.4
+-----------
- Began using the PostgreSQL LISTEN and NOTIFY statements as a shortcut
for invalidation polling.
@@ -230,11 +243,13 @@
Version 0.3
+-----------
- Made compatible with Zope 3, although an undo bug apparently remains.
Version 0.2
+-----------
- Fixed concurrent commits, which were generating deadlocks. Fixed by
adding a special table, "commit_lock", which is used for
Modified: relstorage/branches/1.1/README.txt
===================================================================
--- relstorage/branches/1.1/README.txt 2009-01-27 05:46:41 UTC (rev 95069)
+++ relstorage/branches/1.1/README.txt 2009-01-27 05:56:48 UTC (rev 95070)
@@ -1,46 +1,38 @@
-
Overview
========
- RelStorage is a storage implementation for ZODB that stores pickles in a
+RelStorage is a storage implementation for ZODB that stores pickles in a
relational database. PostgreSQL 8.1 and above (via psycopg2), MySQL 5.0.x (via
-MySQLdb), and Oracle 10g (via cx_Oracle) are currently supported.
+MySQLdb 1.2.2 and above), and Oracle 10g (via cx_Oracle) are currently
+supported. RelStorage replaces the PGStorage project.
- RelStorage replaces the PGStorage project.
- See:
-
- http://wiki.zope.org/ZODB/RelStorage (wiki)
- http://shane.willowrise.com/ (blog)
- http://pypi.python.org/pypi/RelStorage (PyPI entry and downloads)
-
-
Features
========
- * It is a drop-in replacement for FileStorage and ZEO.
- * There is a simple way to convert FileStorage to RelStorage and back again.
-You can also convert a RelStorage instance to a different relational database.
- * Designed for high volume sites: multiple ZODB instances can share the same
-database. This is similar to ZEO, but RelStorage does not require ZEO.
- * According to some tests, RelStorage handles high concurrency better than
-the standard combination of ZEO and FileStorage.
- * Whereas FileStorage takes longer to start as the database grows due to an
-in-memory index of all objects, RelStorage starts quickly regardless of
-database size.
- * Supports undo and packing.
- * Free, open source (ZPL 2.1)
+* It is a drop-in replacement for FileStorage and ZEO.
+* There is a simple way to convert FileStorage to RelStorage and back again.
+ You can also convert a RelStorage instance to a different relational database.
+* Designed for high volume sites: multiple ZODB instances can share the same
+ database. This is similar to ZEO, but RelStorage does not require ZEO.
+* According to some tests, RelStorage handles high concurrency better than
+ the standard combination of ZEO and FileStorage.
+* Whereas FileStorage takes longer to start as the database grows due to an
+ in-memory index of all objects, RelStorage starts quickly regardless of
+ database size.
+* Supports undo and packing.
+* Free, open source (ZPL 2.1)
-Installation in Zope
-====================
+Installation
+============
- You can install RelStorage using easy_install::
+You can install RelStorage using easy_install::
easy_install RelStorage
- If you are not using easy_install (part of the setuptools package), you can
+If you are not using easy_install (part of the setuptools package), you can
get the latest release at PyPI (http://pypi.python.org/pypi/RelStorage), then
place the relstorage package in the lib/python directory of either the
SOFTWARE_HOME or the INSTANCE_HOME. You can do this with the following
@@ -48,21 +40,21 @@
python2.4 setup.py install --install-lib=${INSTANCE_HOME}/lib/python
- Before you can use RelStorage, ZODB must have the invalidation polling patch
-applied. Get it from Subversion (http://svn.zope.org/relstorage/trunk/).
-There are two versions of the patch: one for ZODB 3.7.1 (which is part of Zope
-2.10.5) and one for ZODB 3.8.0 (which is part of Zope 2.11). The patch has no
-effect on ZODB except when using RelStorage. Hopefully, a future release of
-ZODB will include the feature.
+RelStorage requires a version of ZODB with the invalidation polling patch
+applied. You can get versions of ZODB with the patch already applied here:
- You need the Python database adapter that corresponds with your database.
+ http://packages.willowrise.org
+
+The patches are also included in the source distribution of RelStorage.
+
+You need the Python database adapter that corresponds with your database.
Install psycopg2, MySQLdb 1.2.2+, or cx_Oracle 4.3+. Note that Debian Etch
ships MySQLdb 1.2.1, but that version has a bug in BLOB handling that manifests
itself only with certain character set configurations. MySQLdb 1.2.2 fixes the
bug. Also, MySQL 5.1.23 has major bugs that lead to loss of data when packing,
so MySQL 5.1 is not recommended at this time.
- Finally, modify etc/zope.conf of your Zope instance. Remove the main mount
+Finally, modify etc/zope.conf of your Zope instance. Remove the main mount
point and add one of the following blocks. For PostgreSQL::
%import relstorage
@@ -76,7 +68,7 @@
</relstorage>
</zodb_db>
- For MySQL::
+For MySQL::
%import relstorage
<zodb_db main>
@@ -90,7 +82,7 @@
</relstorage>
</zodb_db>
- For Oracle (10g XE in this example)::
+For Oracle (10g XE in this example)::
%import relstorage
<zodb_db main>
@@ -107,67 +99,87 @@
Migrating from FileStorage
==========================
- You can convert a FileStorage instance to RelStorage and back using a utility
+You can convert a FileStorage instance to RelStorage and back using a utility
called ZODBConvert. See http://wiki.zope.org/ZODB/ZODBConvert .
Migrating from PGStorage
========================
- The following script migrates your database from PGStorage to RelStorage 1.0
+The following script migrates your database from PGStorage to RelStorage 1.0
beta:
migrate.sql_
- .. _migrate.sql:
-http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate.sql
+ .. _migrate.sql: http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate.sql
- After you do this, you still need to migrate from 1.0 beta to the latest
+After you do this, you still need to migrate from 1.0 beta to the latest
release.
Migrating to a new version of RelStorage
========================================
- Sometimes RelStorage needs a schema modification along with a software
+Sometimes RelStorage needs a schema modification along with a software
upgrade. Hopefully, this will not often be necessary.
- To migrate from version 1.0 beta to version 1.0c1, see:
+To migrate from version 1.1.1 to version 1.1.2, see:
- migrate-1.0-beta.txt_
+ migrate-to-1.1.2.txt_
- .. _migrate-1.0-beta.txt:
-http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate-1.0-beta.txt
+ .. _migrate-to-1.1.2.txt: http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate-to-1.1.2.txt
- To migrate from version 1.0.1 to version 1.1, see:
+To migrate from version 1.1 to version 1.1.1, see:
- migrate-1.0.1.txt_
+ migrate-to-1.1.1.txt_
- .. _migrate-1.0.1.txt:
-http://svn.zope.org/*checkout*/relstorage/branches/1.1/notes/migrate-1.0.1.txt
+ .. _migrate-to-1.1.1.txt: http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate-to-1.1.1.txt
+To migrate from version 1.0.1 to version 1.1, see:
+ migrate-to-1.1.txt_
+
+ .. _migrate-to-1.1.txt: http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate-to-1.1.txt
+
+To migrate from version 1.0 beta to version 1.0c1 through 1.0.1, see:
+
+ migrate-to-1.0.txt_
+
+ .. _migrate-to-1.0.txt: http://svn.zope.org/*checkout*/relstorage/trunk/notes/migrate-to-1.0.txt
+
+
Optional Features
=================
- Specify these options in zope.conf.
+Specify these options in zope.conf, as parameters for the RelStorage
+constructor, or as attributes of a relstorage.Options instance.
+In the latter two cases, use underscores instead of dashes in the
+parameter names.
- poll-interval
+poll-interval
- Defer polling the database for the specified maximum time interval.
- Set to 0 (the default) to always poll. Fractional seconds are
- allowed.
+ Defer polling the database for the specified maximum time interval,
+ in seconds. Set to 0 (the default) to always poll. Fractional
+ seconds are allowed. Use this to lighten the database load on
+ servers with high read volume and low write volume.
- Use this to lighten the database load on servers with high read
- volume and low write volume. A setting of 1-5 seconds is sufficient
- for most systems.
+ The poll-interval option works best in conjunction with
+ the cache-servers option. If both are enabled, RelStorage will
+ poll a single cache key for changes on every request.
+ The database will not be polled unless the cache indicates
+ there have been changes, or the timeout specified by poll-interval
+ has expired. This configuration keeps clients fully up to date,
+ while removing much of the polling burden from the database.
+ A good cluster configuration is to use memcache servers
+ and a high poll-interval (say, 60 seconds).
- While this setting should not affect database integrity,
- it increases the probability of basing transactions on stale data,
- leading to conflicts. Thus a nonzero setting can hurt
- the performance of servers with high write volume.
+ This option can be used without the cache-servers option,
+ but a large poll-interval without cache-servers increases the
+ probability of basing transactions on stale data, which does not
+ affect database consistency, but does increase the probability
+ of conflict errors, leading to low performance.
- pack-gc
+pack-gc
If pack-gc is false, pack operations do not perform
garbage collection. Garbage collection is enabled by default.
@@ -181,15 +193,23 @@
Disabling garbage collection is also a hack that ensures
inter-database references never break.
- pack-batch-timeout
+pack-dry-run
+ If pack-dry-run is true, pack operations perform a full analysis
+ of what to pack, but no data is actually removed. After a dry run,
+ the pack_object, pack_state, and pack_state_tid tables are filled
+ with the list of object states and objects that would have been
+ removed.
+
+pack-batch-timeout
+
Packing occurs in batches of transactions; this specifies the
timeout in seconds for each batch. Note that some database
configurations have unpredictable I/O performance
and might stall much longer than the timeout.
The default timeout is 5.0 seconds.
- pack-duty-cycle
+pack-duty-cycle
After each batch, the pack code pauses for a time to
allow concurrent transactions to commit. The pack-duty-cycle
@@ -203,14 +223,14 @@
The default is 0.5. Raise it to finish packing faster; lower it
to reduce the effect of packing on transaction commit performance.
- pack-max-delay
+pack-max-delay
This specifies a maximum delay between pack batches. Sometimes
the database takes an extra long time to finish a pack batch; at
those times it is useful to cap the delay imposed by the
pack-duty-cycle. The default is 20 seconds.
- cache-servers
+cache-servers
Specifies a list of memcache servers. Enabling memcache integration
is useful if the connection to the relational database has high
@@ -223,58 +243,67 @@
"127.0.0.1:11211" is a common setting. The default is to disable
memcache integration.
- cache-module-name
+cache-module-name
Specifies which Python memcache module to use. The default is
- "memcache", a pure Python module. An alternative module is
- "cmemcache". This setting has no effect unless cache-servers is set.
+ "memcache", a pure Python module. There are several alternative
+ modules available through PyPI. This setting has no effect unless
+ cache-servers is set.
Development
===========
- You can checkout from Subversion using the following command::
+You can checkout from Subversion using the following command::
svn co svn://svn.zope.org/repos/main/relstorage/trunk RelStorage
- You can also browse the code:
+You can also browse the code:
http://svn.zope.org/relstorage/trunk/
- The best place to discuss development of RelStorage is on the zodb-dev
+The best place to discuss development of RelStorage is on the zodb-dev
mailing list.
-Probable FAQs
-==============
+FAQs
+====
- Q: How can I help?
+Q: How can I help improve RelStorage?
A: The best way to help is to test and to provide database-specific
-expertise. Ask questions about RelStorage on the zodb-dev mailing list.
+ expertise. Ask questions about RelStorage on the zodb-dev mailing list.
- Q: Can I perform SQL queries on the data in the database?
+Q: Can I perform SQL queries on the data in the database?
A: No. Like FileStorage and DirectoryStorage, RelStorage stores the data
-as pickles, making it hard for anything but ZODB to interpret the data. An
-earlier project called Ape attempted to store data in a truly relational way,
-but it turned out that Ape worked too much against ZODB principles and
-therefore could not be made reliable enough for production use. RelStorage, on
-the other hand, is much closer to an ordinary ZODB storage, and is therefore
-much safer for production use.
+ as pickles, making it hard for anything but ZODB to interpret the data. An
+ earlier project called Ape attempted to store data in a truly relational
+ way, but it turned out that Ape worked too much against ZODB principles and
+ therefore could not be made reliable enough for production use. RelStorage,
+ on the other hand, is much closer to an ordinary ZODB storage, and is
+ therefore much safer for production use.
- Q: How does RelStorage performance compare with FileStorage?
+Q: How does RelStorage performance compare with FileStorage?
A: According to benchmarks, RelStorage with PostgreSQL is often faster than
-FileStorage, especially under high concurrency.
+ FileStorage, especially under high concurrency.
- Q: Why should I choose RelStorage?
+Q: Why should I choose RelStorage?
A: Because RelStorage is a fairly small layer that builds on world-class
-databases. These databases have proven reliability and scalability, along with
-numerous support options.
+ databases. These databases have proven reliability and scalability, along
+ with numerous support options.
- Q: Can RelStorage replace ZRS (Zope Replication Services)?
+Q: Can RelStorage replace ZRS (Zope Replication Services)?
A: In theory, yes. With RelStorage, you can use the replication features
-native to your database. However, this capability has not yet been tested.
+ native to your database. However, this capability has not yet been tested.
+
+
+Project URLs
+============
+
+* http://wiki.zope.org/ZODB/RelStorage (wiki)
+* http://shane.willowrise.com/ (blog)
+* http://pypi.python.org/pypi/RelStorage (PyPI entry and downloads)
Modified: relstorage/branches/1.1/setup.py
===================================================================
--- relstorage/branches/1.1/setup.py 2009-01-27 05:46:41 UTC (rev 95069)
+++ relstorage/branches/1.1/setup.py 2009-01-27 05:56:48 UTC (rev 95070)
@@ -11,24 +11,15 @@
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
-"""A backend for ZODB that stores pickles in a relational database.
+"""A backend for ZODB that stores pickles in a relational database."""
-This is designed to be a drop-in replacement for the standard ZODB
-combination of FileStorage and ZEO. Multiple ZODB clients can
-share the same database without any additional configuration.
-Supports undo, historical database views, packing, and lossless
-migration between FileStorage and RelStorage instances.
+VERSION = "1.1.2dev"
-The supported relational databases are PostgreSQL 8.1 and above
-(using the psycopg2 Python module), MySQL 5.0 and above (using the
-MySQLdb 1.2.2 Python module), and Oracle 10g (using cx_Oracle 4.3).
+# The choices for the Trove Development Status line:
+# Development Status :: 5 - Production/Stable
+# Development Status :: 4 - Beta
+# Development Status :: 3 - Alpha
-A small patch to ZODB is required. See the patch files distributed
-with RelStorage.
-"""
-
-VERSION = "1.1.2dev"
-
classifiers = """\
Development Status :: 5 - Production/Stable
Intended Audience :: Developers
@@ -40,6 +31,7 @@
Operating System :: Unix
"""
+import os
try:
from setuptools import setup
except ImportError:
@@ -64,6 +56,11 @@
doclines = __doc__.split("\n")
+def read_file(*path):
+ base_dir = os.path.dirname(__file__)
+ file_path = (base_dir, ) + tuple(path)
+ return file(os.path.join(*file_path)).read()
+
setup(
name="RelStorage",
version=VERSION,
@@ -78,6 +75,10 @@
platforms=["any"],
description=doclines[0],
classifiers=filter(None, classifiers.split("\n")),
- long_description = "\n".join(doclines[2:]),
+ long_description = (
+ read_file("README.txt") + "\n\n" +
+ "Changes\n" +
+ "=======\n\n" +
+ read_file("CHANGES.txt")),
**setuptools_args
)
More information about the Checkins
mailing list