[Checkins] SVN: relstorage/trunk/ Fixed compatibility with PostgreSQL 9.0, which is capable of
Shane Hathaway
shane at hathawaymix.org
Mon Feb 28 18:01:50 EST 2011
Log message for revision 120614:
Fixed compatibility with PostgreSQL 9.0, which is capable of
returning a new 'hex' type to the client. Some builds of psycopg2
return garbage or raise an error when they see the new type. The fix
was to encode more SQL query responses using base 64.
Also added credit for Martijn. Awesome work.
Changed:
U relstorage/trunk/CHANGES.txt
U relstorage/trunk/relstorage/adapters/dbiter.py
U relstorage/trunk/relstorage/adapters/mysql.py
U relstorage/trunk/relstorage/adapters/oracle.py
U relstorage/trunk/relstorage/adapters/postgresql.py
U relstorage/trunk/relstorage/adapters/scriptrunner.py
A relstorage/trunk/relstorage/tests/README-15.txt
-=-
Modified: relstorage/trunk/CHANGES.txt
===================================================================
--- relstorage/trunk/CHANGES.txt 2011-02-28 21:15:06 UTC (rev 120613)
+++ relstorage/trunk/CHANGES.txt 2011-02-28 23:01:50 UTC (rev 120614)
@@ -1,30 +1,40 @@
Next Release
------------
-- Added more feedback to the packing process. It'll now report
- during batch commit how much of the total work has been completed,
- but at most every .1% of the total number of transactions or objects to
- process.
+- Better packing based on experience with large databases. Thanks
+ to Martijn Pieters!
-- Renamed the --dry-run option to --prepack and added a --use-prepack-state
- to zodbpack. With these 2 options the pre-pack and pack phases can be run
- separately, allowing re-use of the pre-pack analysis data or even delegating
- the pre-pack phase off to a separate server.
+ - Added more feedback to the packing process. It'll now report
+ during batch commit how much of the total work has been
+ completed, but at most every .1% of the total number of
+ transactions or objects to process.
-- Replaced the packing duty cycle with a nowait locking strategy. The pack
- operation will now request the commit lock but pauses if it is already taken.
- It releases the lock after every batch (defaulting to 1 second processing).
- This makes the packing process faster while at the same time yielding to
- regular ZODB commits when busy.
+ - Renamed the --dry-run option to --prepack and added a
+ --use-prepack-state to zodbpack. With these 2 options the
+ pre-pack and pack phases can be run separately, allowing re-use
+ of the pre-pack analysis data or even delegating the pre-pack
+ phase off to a separate server.
-- Do not hold the commit lock during pack cleanup while deleting rows from
- the object reference tables; these tables are pack-specific and regular
- ZODB commits never touch these.
+ - Replaced the packing duty cycle with a nowait locking strategy.
+ The pack operation will now request the commit lock but pauses if
+ it is already taken. It releases the lock after every batch
+ (defaulting to 1 second processing). This makes the packing
+ process faster while at the same time yielding to regular ZODB
+ commits when busy.
-- Add an option to control schema creation / updating on startup. Setting the
- ``create-schema`` option to false will let you use RelStorage without a
- schema update.
+ - Do not hold the commit lock during pack cleanup while deleting
+ rows from the object reference tables; these tables are
+ pack-specific and regular ZODB commits never touch these.
+- Add an option to control schema creation / updating on startup.
+ Setting the ``create-schema`` option to false will let you use
+ RelStorage without a schema update.
+
+- Fixed compatibility with PostgreSQL 9.0, which is capable of
+ returning a new 'hex' type to the client. Some builds of psycopg2
+ return garbage or raise an error when they see the new type. The fix
+ was to encode more SQL query responses using base 64.
+
1.5.0b1 (2011-02-05)
--------------------
Modified: relstorage/trunk/relstorage/adapters/dbiter.py
===================================================================
--- relstorage/trunk/relstorage/adapters/dbiter.py 2011-02-28 21:15:06 UTC (rev 120613)
+++ relstorage/trunk/relstorage/adapters/dbiter.py 2011-02-28 23:01:50 UTC (rev 120614)
@@ -12,14 +12,17 @@
#
##############################################################################
+from base64 import decodestring
from relstorage.adapters.interfaces import IDatabaseIterator
from zope.interface import implements
+
class DatabaseIterator(object):
"""Abstract base class for database iteration.
"""
- def __init__(self, runner):
+ def __init__(self, database_name, runner):
+ self.use_base64 = (database_name == 'postgresql')
self.runner = runner
def iter_objects(self, cursor, tid):
@@ -27,17 +30,27 @@
Yields (oid, prev_tid, state) for each object state.
"""
- stmt = """
- SELECT zoid, state
- FROM object_state
- WHERE tid = %(tid)s
- ORDER BY zoid
- """
+ if self.use_base64:
+ stmt = """
+ SELECT zoid, encode(state, 'base64')
+ FROM object_state
+ WHERE tid = %(tid)s
+ ORDER BY zoid
+ """
+ else:
+ stmt = """
+ SELECT zoid, state
+ FROM object_state
+ WHERE tid = %(tid)s
+ ORDER BY zoid
+ """
self.runner.run_script_stmt(cursor, stmt, {'tid': tid})
for oid, state in cursor:
if hasattr(state, 'read'):
# Oracle
state = state.read()
+ if state is not None and self.use_base64:
+ state = decodestring(state)
yield oid, state
@@ -50,20 +63,27 @@
Each row begins with (tid, username, description, extension)
and may have other columns.
"""
+ use_base64 = self.use_base64
for row in cursor:
tid, username, description, ext = row[:4]
if username is None:
username = ''
else:
username = str(username)
+ if use_base64:
+ username = decodestring(username)
if description is None:
description = ''
else:
description = str(description)
+ if use_base64:
+ description = decodestring(description)
if ext is None:
ext = ''
else:
ext = str(ext)
+ if use_base64:
+ ext = decodestring(ext)
yield (tid, username, description, ext) + tuple(row[4:])
@@ -73,13 +93,23 @@
Skips packed transactions.
Yields (tid, username, description, extension) for each transaction.
"""
- stmt = """
- SELECT tid, username, description, extension
- FROM transaction
- WHERE packed = %(FALSE)s
- AND tid != 0
- ORDER BY tid DESC
- """
+ if self.use_base64:
+ stmt = """
+ SELECT tid, encode(username, 'base64'),
+ encode(description, 'base64'), encode(extension, 'base64')
+ FROM transaction
+ WHERE packed = %(FALSE)s
+ AND tid != 0
+ ORDER BY tid DESC
+ """
+ else:
+ stmt = """
+ SELECT tid, username, description, extension
+ FROM transaction
+ WHERE packed = %(FALSE)s
+ AND tid != 0
+ ORDER BY tid DESC
+ """
self.runner.run_script_stmt(cursor, stmt)
return self._transaction_iterator(cursor)
@@ -91,12 +121,21 @@
Yields (tid, username, description, extension, packed)
for each transaction.
"""
- stmt = """
- SELECT tid, username, description, extension,
- CASE WHEN packed = %(TRUE)s THEN 1 ELSE 0 END
- FROM transaction
- WHERE tid >= 0
- """
+ if self.use_base64:
+ stmt = """
+ SELECT tid, encode(username, 'base64'),
+ encode(description, 'base64'), encode(extension, 'base64'),
+ CASE WHEN packed = %(TRUE)s THEN 1 ELSE 0 END
+ FROM transaction
+ WHERE tid >= 0
+ """
+ else:
+ stmt = """
+ SELECT tid, username, description, extension,
+ CASE WHEN packed = %(TRUE)s THEN 1 ELSE 0 END
+ FROM transaction
+ WHERE tid >= 0
+ """
if start is not None:
stmt += " AND tid >= %(min_tid)s"
if stop is not None:
@@ -121,8 +160,17 @@
if not cursor.fetchall():
raise KeyError(oid)
- stmt = """
- SELECT tid, username, description, extension, %(OCTET_LENGTH)s(state)
+ if self.use_base64:
+ stmt = """
+ SELECT tid, encode(username, 'base64'),
+ encode(description, 'base64'), encode(extension, 'base64'),
+ state_size
+ """
+ else:
+ stmt = """
+ SELECT tid, username, description, extension, state_size
+ """
+ stmt += """
FROM transaction
JOIN object_state USING (tid)
WHERE zoid = %(oid)s
@@ -173,7 +221,7 @@
for each modification.
"""
stmt = """
- SELECT tid, %(OCTET_LENGTH)s(state)
+ SELECT tid, state_size
FROM object_state
WHERE zoid = %(oid)s
"""
Modified: relstorage/trunk/relstorage/adapters/mysql.py
===================================================================
--- relstorage/trunk/relstorage/adapters/mysql.py 2011-02-28 21:15:06 UTC (rev 120613)
+++ relstorage/trunk/relstorage/adapters/mysql.py 2011-02-28 23:01:50 UTC (rev 120614)
@@ -139,6 +139,7 @@
options=options,
)
self.dbiter = HistoryPreservingDatabaseIterator(
+ database_name='mysql',
runner=self.runner,
)
else:
@@ -149,6 +150,7 @@
options=options,
)
self.dbiter = HistoryFreeDatabaseIterator(
+ database_name='mysql',
runner=self.runner,
)
Modified: relstorage/trunk/relstorage/adapters/oracle.py
===================================================================
--- relstorage/trunk/relstorage/adapters/oracle.py 2011-02-28 21:15:06 UTC (rev 120613)
+++ relstorage/trunk/relstorage/adapters/oracle.py 2011-02-28 23:01:50 UTC (rev 120614)
@@ -134,6 +134,7 @@
options=options,
)
self.dbiter = HistoryPreservingDatabaseIterator(
+ database_name='oracle',
runner=self.runner,
)
else:
@@ -144,6 +145,7 @@
options=options,
)
self.dbiter = HistoryFreeDatabaseIterator(
+ database_name='oracle',
runner=self.runner,
)
Modified: relstorage/trunk/relstorage/adapters/postgresql.py
===================================================================
--- relstorage/trunk/relstorage/adapters/postgresql.py 2011-02-28 21:15:06 UTC (rev 120613)
+++ relstorage/trunk/relstorage/adapters/postgresql.py 2011-02-28 23:01:50 UTC (rev 120614)
@@ -104,6 +104,7 @@
options=options,
)
self.dbiter = HistoryPreservingDatabaseIterator(
+ database_name='postgresql',
runner=self.runner,
)
else:
@@ -114,6 +115,7 @@
options=options,
)
self.dbiter = HistoryFreeDatabaseIterator(
+ database_name='postgresql',
runner=self.runner,
)
Modified: relstorage/trunk/relstorage/adapters/scriptrunner.py
===================================================================
--- relstorage/trunk/relstorage/adapters/scriptrunner.py 2011-02-28 21:15:06 UTC (rev 120613)
+++ relstorage/trunk/relstorage/adapters/scriptrunner.py 2011-02-28 23:01:50 UTC (rev 120614)
@@ -28,7 +28,6 @@
script_vars = {
'TRUE': 'TRUE',
'FALSE': 'FALSE',
- 'OCTET_LENGTH': 'OCTET_LENGTH',
'TRUNCATE': 'TRUNCATE',
'oid': '%(oid)s',
'tid': '%(tid)s',
@@ -95,7 +94,6 @@
script_vars = {
'TRUE': "'Y'",
'FALSE': "'N'",
- 'OCTET_LENGTH': 'LENGTH',
'TRUNCATE': 'TRUNCATE TABLE',
'oid': ':oid',
'tid': ':tid',
Added: relstorage/trunk/relstorage/tests/README-15.txt
===================================================================
--- relstorage/trunk/relstorage/tests/README-15.txt (rev 0)
+++ relstorage/trunk/relstorage/tests/README-15.txt 2011-02-28 23:01:50 UTC (rev 120614)
@@ -0,0 +1,63 @@
+
+Running Tests
+=============
+
+To run tests of both RelStorage 1.4 and 1.5 on the same box, you need
+a set of databases for each version, since they have different schemas.
+Do this after following the instructions in README.txt:
+
+
+PostgreSQL
+----------
+
+Execute the following using the ``psql`` command::
+
+ CREATE DATABASE relstorage15test OWNER relstoragetest;
+ CREATE DATABASE relstorage15test2 OWNER relstoragetest;
+ CREATE DATABASE relstorage15test_hf OWNER relstoragetest;
+ CREATE DATABASE relstorage15test2_hf OWNER relstoragetest;
+
+Also, add the following lines to the top of pg_hba.conf (if you put
+them at the bottom, they may be overridden by other parameters)::
+
+ local relstorage15test relstoragetest md5
+ local relstorage15test2 relstoragetest md5
+ local relstorage15test_hf relstoragetest md5
+ local relstorage15test2_hf relstoragetest md5
+ host relstorage15test relstoragetest 127.0.0.1/32 md5
+ host relstorage15test_hf relstoragetest 127.0.0.1/32 md5
+
+
+
+MySQL
+-----
+
+ CREATE DATABASE relstorage15test;
+ GRANT ALL ON relstorage15test.* TO 'relstoragetest'@'localhost';
+ CREATE DATABASE relstorage15test2;
+ GRANT ALL ON relstorage15test2.* TO 'relstoragetest'@'localhost';
+ CREATE DATABASE relstorage15test_hf;
+ GRANT ALL ON relstorage15test_hf.* TO 'relstoragetest'@'localhost';
+ CREATE DATABASE relstorage15test2_hf;
+ GRANT ALL ON relstorage15test2_hf.* TO 'relstoragetest'@'localhost';
+ FLUSH PRIVILEGES;
+
+
+Oracle
+------
+
+Using ``sqlplus`` with ``SYS`` privileges, execute the
+following::
+
+ CREATE USER relstorage15test IDENTIFIED BY relstoragetest;
+ GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstorage15test;
+ GRANT EXECUTE ON DBMS_LOCK TO relstorage15test;
+ CREATE USER relstoragetest2 IDENTIFIED BY relstorage15test;
+ GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstorage15test2;
+ GRANT EXECUTE ON DBMS_LOCK TO relstorage15test2;
+ CREATE USER relstoragetest_hf IDENTIFIED BY relstorage15test;
+ GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstorage15test_hf;
+ GRANT EXECUTE ON DBMS_LOCK TO relstorage15test_hf;
+ CREATE USER relstoragetest2_hf IDENTIFIED BY relstorage15test;
+ GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstorage15test2_hf;
+ GRANT EXECUTE ON DBMS_LOCK TO relstorage15test2_hf;
More information about the checkins
mailing list