[Checkins] SVN: relstorage/trunk/ Moved the keep-history and replica options outside the adapter-specific
Shane Hathaway
shane at hathawaymix.org
Sat Oct 3 17:54:16 EDT 2009
Log message for revision 104768:
Moved the keep-history and replica options outside the adapter-specific
configuration. Also documented the read-only and name options.
Changed:
U relstorage/trunk/CHANGES.txt
U relstorage/trunk/README.txt
U relstorage/trunk/relstorage/adapters/connmanager.py
U relstorage/trunk/relstorage/adapters/interfaces.py
U relstorage/trunk/relstorage/adapters/mysql.py
U relstorage/trunk/relstorage/adapters/oracle.py
U relstorage/trunk/relstorage/adapters/postgresql.py
A relstorage/trunk/relstorage/adapters/replica.py
U relstorage/trunk/relstorage/adapters/tests/test_connmanager.py
A relstorage/trunk/relstorage/adapters/tests/test_replica.py
U relstorage/trunk/relstorage/component.xml
U relstorage/trunk/relstorage/config.py
A relstorage/trunk/relstorage/options.py
U relstorage/trunk/relstorage/storage.py
U relstorage/trunk/relstorage/tests/testmysql.py
U relstorage/trunk/relstorage/tests/testoracle.py
U relstorage/trunk/relstorage/tests/testpostgresql.py
-=-
Modified: relstorage/trunk/CHANGES.txt
===================================================================
--- relstorage/trunk/CHANGES.txt 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/CHANGES.txt 2009-10-03 21:54:16 UTC (rev 104768)
@@ -2,17 +2,17 @@
Unreleased
----------
-- Added the keep-history option to the database adapters. Set it
- to false to keep no history. (Packing is still required for
- garbage collection and blob deletion.)
+- Added the keep-history option. Set it to false to keep no history.
+ (Packing is still required for garbage collection and blob deletion.)
-- Added the replica-conf option to the database adapters. Set it
+- Added the replica-conf and replica-timeout options. Set replica-conf
to a filename containing the location of database replicas. Changes
to the file take effect at transaction boundaries.
-- Copied the ZConfig documentation of the storage options into README.
+- Expanded the option documentation in README.txt.
- Renamed relstorage.py to storage.py to overcome import issues.
+ Also moved the Options class to options.py.
- Updated the patch for ZODB 3.7 and 3.8 to fix an issue with
blobs and subtransactions.
Modified: relstorage/trunk/README.txt
===================================================================
--- relstorage/trunk/README.txt 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/README.txt 2009-10-03 21:54:16 UTC (rev 104768)
@@ -242,12 +242,61 @@
``relstorage.storage.Options`` instance. In the latter two cases, use
underscores instead of dashes in the parameter names.
+``name``
+ The name of the storage. Defaults to a descriptive name
+ that includes most of the adapter configuration parameters
+ except the database password.
+
+``read-only``
+ If true, only reads may be executed against the storage. Note
+ that the "pack" operation is not considered a write operation
+ and is still allowed on a read-only filestorage.
+
``blob-dir``
If supplied, the storage will provide blob support and this
is the name of a directory to hold blob data. The directory will
be created if it doeesn't exist. If no value (or an empty value)
is provided, then no blob support will be provided.
+``keep-history``
+ If this parameter is set to true (the default), the adapter
+ will create and use a history-preserving database schema
+ like FileStorage. A history-preserving schema supports
+ ZODB-level undo, but also grows more quickly and requires extensive
+ packing on a regular basis.
+
+ If this parameter is set to false, the adapter will create and
+ use a history-free database schema. Undo will not be supported,
+ but the database will not grow as quickly. The database will
+ still require regular garbage collection (which is accessible
+ through the database pack mechanism.)
+
+ This parameter must not change once the database schema has
+ been installed, because the schemas for history-preserving and
+ history-free storage are different. If you want to convert
+ between a history-preserving and a history-free database, use
+ the ``zodbconvert`` utility to copy to a new database.
+
+``replica-conf``
+ If this parameter is provided, it specifies a text file that
+ contains a list of database replicas the adapter can choose
+ from. For MySQL and PostgreSQL, put in the replica file a list
+ of ``host:port`` or ``host`` values, one per line. For Oracle,
+ put in a list of DSN values. Blank lines and lines starting
+ with ``#`` are ignored.
+
+ The adapter prefers the first replica specified in the file. If
+ the first is not available, the adapter automatically tries the
+ rest of the replicas, in order. If the file changes, the
+ adapter will drop existing SQL database connections and make
+ new connections when ZODB starts a new transaction.
+
+``replica-timeout``
+ If this parameter has a nonzero value, when the adapter selects
+ a replica other than the primary replica, the adapter will
+ try to revert to the primary replica after the specified
+ timeout (in seconds). The default is 600, meaning 10 minutes.
+
``poll-interval``
Defer polling the database for the specified maximum time interval,
in seconds. Set to 0 (the default) to always poll. Fractional
@@ -337,45 +386,6 @@
Adapter Options
===============
-Common Adapter Options
-----------------------
-
-All current RelStorage adapters support the following options.
-
-``keep-history``
- If this parameter is set to true (the default), the adapter
- will create and use a history-preserving database schema
- like FileStorage. A history-preserving schema supports
- ZODB-level undo, but also grows more quickly and requires extensive
- packing on a regular basis.
-
- If this parameter is set to false, the adapter will create and
- use a history-free database schema. Undo will not be supported,
- but the database will not grow as quickly. The database will
- still require regular garbage collection (which is accessible
- through the database pack mechanism.)
-
- This parameter must not change once the database schema has
- been installed, because the schemas for history-preserving and
- history-free storage are different. If you want to convert
- between a history-preserving and a history-free database, use
- the ``zodbconvert`` utility to copy to a new database.
-
-``replica-conf``
- If this parameter is provided, it specifies a text file that
- contains a list of database replicas this adapter can choose
- from. For MySQL and PostgreSQL, put in the replica file a list
- of ``host:port`` or ``host`` values, one per line. For Oracle,
- put in a list of DSN values. Blank lines and lines starting
- with ``#`` are ignored.
-
- The adapter prefers the first replica specified in the file. If
- the first is not available, the adapter automatically tries the
- rest of the replicas, in order. If the file changes, the
- adapter will drop existing SQL database connections and make
- new connections when ZODB starts a new transaction.
-
-
PostgreSQL Adapter Options
--------------------------
Modified: relstorage/trunk/relstorage/adapters/connmanager.py
===================================================================
--- relstorage/trunk/relstorage/adapters/connmanager.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/adapters/connmanager.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -14,9 +14,8 @@
from relstorage.adapters.interfaces import IConnectionManager
from relstorage.adapters.interfaces import ReplicaClosedException
+from relstorage.adapters.replica import ReplicaSelector
from zope.interface import implements
-import os
-import time
class AbstractConnectionManager(object):
@@ -38,11 +37,12 @@
# will be called whenever a store cursor is opened or rolled back.
on_store_opened = None
- def __init__(self, replica_conf=None):
- if replica_conf:
- self.replicas = ReplicaSelector(replica_conf)
+ def __init__(self, options=None):
+ # options is a relstorage.options.Options instance
+ if options is not None and options.replica_conf:
+ self.replica_selector = ReplicaSelector(options)
else:
- self.replicas = None
+ self.replica_selector = None
def set_on_store_opened(self, f):
"""Set the on_store_opened hook"""
@@ -88,12 +88,18 @@
def restart_load(self, conn, cursor):
"""Reinitialize a connection for loading objects."""
- if self.replicas is not None:
- if conn.replica != self.replicas.current():
+ self.check_replica(conn, cursor)
+ conn.rollback()
+
+ def check_replica(self, conn, cursor):
+ """Raise an exception if the connection belongs to an old replica"""
+ if self.replica_selector is not None:
+ current = self.replica_selector.current()
+ if conn.replica != current:
# Prompt the change to a new replica by raising an exception.
self.close(conn, cursor)
- raise ReplicaClosedException()
- conn.rollback()
+ raise ReplicaClosedException(
+ "Switched replica from %s to %s" % (conn.replica, current))
def open_for_store(self):
"""Open and initialize a connection for storing objects.
@@ -111,11 +117,7 @@
def restart_store(self, conn, cursor):
"""Reuse a store connection."""
- if self.replicas is not None:
- if conn.replica != self.replicas.current():
- # Prompt the change to a new replica by raising an exception.
- self.close(conn, cursor)
- raise ReplicaClosedException()
+ self.check_replica(conn, cursor)
conn.rollback()
if self.on_store_opened is not None:
self.on_store_opened(cursor, restart=True)
@@ -126,95 +128,3 @@
"""
return self.open()
-
-class ReplicaSelector(object):
-
- def __init__(self, replica_conf, alt_timeout=600):
- self.replica_conf = replica_conf
- self.alt_timeout = alt_timeout
- self._read_config()
- self._select(0)
- self._iterating = False
- self._skip_index = None
-
- def _read_config(self):
- self._config_modified = os.path.getmtime(self.replica_conf)
- self._config_checked = time.time()
- f = open(self.replica_conf, 'r')
- try:
- lines = f.readlines()
- finally:
- f.close()
- replicas = []
- for line in lines:
- line = line.strip()
- if not line or line.startswith('#'):
- continue
- replicas.append(line)
- if not replicas:
- raise IndexError(
- "No replicas specified in %s" % self.replica_conf)
- self._replicas = replicas
-
- def _is_config_modified(self):
- now = time.time()
- if now < self._config_checked + 1:
- # don't check the last mod time more often than once per second
- return False
- self._config_checked = now
- t = os.path.getmtime(self.replica_conf)
- return t != self._config_modified
-
- def _select(self, index):
- self._current_replica = self._replicas[index]
- self._current_index = index
- if index > 0 and self.alt_timeout:
- self._expiration = time.time() + self.alt_timeout
- else:
- self._expiration = None
-
- def current(self):
- """Get the current replica."""
- self._iterating = False
- if self._is_config_modified():
- self._read_config()
- self._select(0)
- elif self._expiration is not None and time.time() >= self._expiration:
- self._select(0)
- return self._current_replica
-
- def next(self):
- """Return the next replica to try.
-
- Return None if there are no more replicas defined.
- """
- if self._is_config_modified():
- # Start over even if iteration was already in progress.
- self._read_config()
- self._select(0)
- self._skip_index = None
- self._iterating = True
- elif not self._iterating:
- # Start iterating.
- self._skip_index = self._current_index
- i = 0
- if i == self._skip_index:
- i = 1
- if i >= len(self._replicas):
- # There are no more replicas to try.
- self._select(0)
- return None
- self._select(i)
- self._iterating = True
- else:
- # Continue iterating.
- i = self._current_index + 1
- if i == self._skip_index:
- i += 1
- if i >= len(self._replicas):
- # There are no more replicas to try.
- self._select(0)
- return None
- self._select(i)
-
- return self._current_replica
Modified: relstorage/trunk/relstorage/adapters/interfaces.py
===================================================================
--- relstorage/trunk/relstorage/adapters/interfaces.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/adapters/interfaces.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -102,6 +102,24 @@
"""
+class IReplicaSelector(Interface):
+ """Selects a database replica"""
+
+ def current():
+ """Get the current replica.
+
+ Return a string. For PostgreSQL and MySQL, the string is
+ either a host:port specification or host name. For Oracle,
+ the string is a DSN.
+ """
+
+ def next():
+ """Return the next replica to try.
+
+ Return None if there are no more replicas defined.
+ """
+
+
class IDatabaseIterator(Interface):
"""Iterate over the available data in the database"""
Modified: relstorage/trunk/relstorage/adapters/mysql.py
===================================================================
--- relstorage/trunk/relstorage/adapters/mysql.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/adapters/mysql.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -87,10 +87,18 @@
"""MySQL adapter for RelStorage."""
implements(IRelStorageAdapter)
- def __init__(self, keep_history=True, **params):
- self.keep_history = keep_history
+ def __init__(self, options=None, **params):
+ # options is a relstorage.options.Options or None
+ self.options = options
+ if options is not None:
+ self.keep_history = options.keep_history
+ else:
+ self.keep_history = True
self._params = params
- self.connmanager = MySQLdbConnectionManager(**params)
+ self.connmanager = MySQLdbConnectionManager(
+ params=params,
+ options=options,
+ )
self.runner = ScriptRunner()
self.locker = MySQLLocker(
keep_history=self.keep_history,
@@ -147,20 +155,21 @@
)
def new_instance(self):
- return MySQLAdapter(keep_history=self.keep_history, **self._params)
+ return MySQLAdapter(options=self.options, **self._params)
def __str__(self):
+ parts = [self.__class__.__name__]
if self.keep_history:
- t = 'history preserving'
+ parts.append('history preserving')
else:
- t = 'history free'
+ parts.append('history free')
p = self._params.copy()
if 'passwd' in p:
del p['passwd']
p = p.items()
p.sort()
- p = ', '.join('%s=%r' % item for item in p)
- return "%s, %s, %s" % (self.__class__.__name__, t, p)
+ parts.extend('%s=%r' % item for item in p)
+ return ", ".join(parts)
class MySQLdbConnectionManager(AbstractConnectionManager):
@@ -171,19 +180,20 @@
disconnected_exceptions = disconnected_exceptions
close_exceptions = close_exceptions
- def __init__(self, replica_conf=None, **params):
+ def __init__(self, params, options=None):
self._orig_params = params.copy()
self._params = self._orig_params
- self._current_replica = None
- super(MySQLdbConnectionManager, self).__init__(
- replica_conf=replica_conf)
+ # _params_derived_from_replica contains the replica that
+ # was used to set self._params.
+ self._params_derived_from_replica = None
+ super(MySQLdbConnectionManager, self).__init__(options)
def _set_params(self, replica):
"""Alter the connection parameters to use the specified replica.
The replica parameter is a string specifying either host or host:port.
"""
- if replica != self._current_replica:
+ if replica != self._params_derived_from_replica:
params = self._orig_params.copy()
if ':' in replica:
host, port = replica.split(':')
@@ -191,13 +201,17 @@
params['port'] = int(port)
else:
params['host'] = replica
- self._current_replica = replica
+ self._params_derived_from_replica = replica
self._params = params
def open(self, transaction_mode="ISOLATION LEVEL READ COMMITTED"):
"""Open a database connection and return (conn, cursor)."""
- if self.replicas is not None:
- self._set_params(self.replicas.current())
+ if self.replica_selector is not None:
+ replica = self.replica_selector.current()
+ self._set_params(replica)
+ else:
+ replica = None
+
while True:
try:
conn = MySQLdb.connect(**self._params)
@@ -208,16 +222,16 @@
cursor.execute(
"SET SESSION TRANSACTION %s" % transaction_mode)
conn.autocommit(False)
- conn.replica = self._current_replica
+ conn.replica = replica
return conn, cursor
except MySQLdb.OperationalError, e:
- if self._current_replica:
+ if replica is not None:
log.warning("Unable to connect to replica %s: %s",
- self._current_replica, e)
+ replica, e)
else:
log.warning("Unable to connect: %s", e)
- if self.replicas is not None:
- replica = self.replicas.next()
+ if self.replica_selector is not None:
+ replica = self.replica_selector.next()
if replica is not None:
# try the new replica
self._set_params(replica)
Modified: relstorage/trunk/relstorage/adapters/oracle.py
===================================================================
--- relstorage/trunk/relstorage/adapters/oracle.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/adapters/oracle.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -52,8 +52,7 @@
"""Oracle adapter for RelStorage."""
implements(IRelStorageAdapter)
- def __init__(self, user, password, dsn, twophase=False,
- keep_history=True, replica_conf=None):
+ def __init__(self, user, password, dsn, twophase=False, options=None):
"""Create an Oracle adapter.
The user, password, and dsn parameters are provided to cx_Oracle
@@ -63,19 +62,23 @@
commit process. This is disabled by default. Even when this option
is disabled, the ZODB two-phase commit is still in effect.
"""
+ # options is a relstorage.options.Options or None
self._user = user
self._password = password
self._dsn = dsn
self._twophase = twophase
- self.keep_history = keep_history
- self.replica_conf = replica_conf
+ self.options = options
+ if options is not None:
+ self.keep_history = options.keep_history
+ else:
+ self.keep_history = True
self.connmanager = CXOracleConnectionManager(
user=user,
password=password,
dsn=dsn,
twophase=twophase,
- replica_conf=replica_conf,
+ options=options,
)
self.runner = CXOracleScriptRunner()
self.locker = OracleLocker(
@@ -146,8 +149,7 @@
password=self._password,
dsn=self._dsn,
twophase=self._twophase,
- keep_history=self.keep_history,
- replica_conf=self.replica_conf,
+ options=self.options,
)
def __str__(self):
@@ -159,7 +161,6 @@
parts.append('user=%r' % self._user)
parts.append('dsn=%r' % self._dsn)
parts.append('twophase=%r' % self._twophase)
- parts.append('replica_conf=%r' % self.replica_conf)
return ", ".join(parts)
@@ -232,19 +233,19 @@
disconnected_exceptions = disconnected_exceptions
close_exceptions = close_exceptions
- def __init__(self, user, password, dsn, twophase, replica_conf=None):
+ def __init__(self, user, password, dsn, twophase, options=None):
self._user = user
self._password = password
self._dsn = dsn
self._twophase = twophase
- super(CXOracleConnectionManager, self).__init__(
- replica_conf=replica_conf)
+ super(CXOracleConnectionManager, self).__init__(options)
def open(self, transaction_mode="ISOLATION LEVEL READ COMMITTED",
twophase=False):
"""Open a database connection and return (conn, cursor)."""
- if self.replicas is not None:
- self._dsn = self.replicas.current()
+ if self.replica_selector is not None:
+ self._dsn = self.replica_selector.current()
+
while True:
try:
kw = {'twophase': twophase} #, 'threaded': True}
@@ -258,8 +259,8 @@
except cx_Oracle.OperationalError, e:
log.warning("Unable to connect to DSN %s: %s", self._dsn, e)
- if self.replicas is not None:
- replica = self.replicas.next()
+ if self.replica_selector is not None:
+ replica = self.replica_selector.next()
if replica is not None:
# try the new replica
self._dsn = replica
@@ -275,14 +276,20 @@
def restart_load(self, conn, cursor):
"""Reinitialize a connection for loading objects."""
- if self.replicas is not None:
- if conn.dsn != self.replicas.current():
- # Prompt the change to a new replica by raising an exception.
- self.close(conn, cursor)
- raise ReplicaClosedException()
+ self.check_replica(conn, cursor)
conn.rollback()
cursor.execute("SET TRANSACTION READ ONLY")
+ def check_replica(self, conn, cursor):
+ """Raise an exception if the connection belongs to an old replica"""
+ if self.replica_selector is not None:
+ current = self.replica_selector.current()
+ if conn.dsn != current:
+ # Prompt the change to a new replica by raising an exception.
+ self.close(conn, cursor)
+ raise ReplicaClosedException(
+ "Switching replica from %s to %s" % (conn.dsn, current))
+
def _set_xid(self, conn, cursor):
"""Set up a distributed transaction"""
stmt = """
@@ -313,11 +320,7 @@
def restart_store(self, conn, cursor):
"""Reuse a store connection."""
- if self.replicas is not None:
- if conn.dsn != self.replicas.current():
- # Prompt the change to a new replica by raising an exception.
- self.close(conn, cursor)
- raise ReplicaClosedException()
+ self.check_replica(conn, cursor)
conn.rollback()
if self._twophase:
self._set_xid(conn, cursor)
Modified: relstorage/trunk/relstorage/adapters/postgresql.py
===================================================================
--- relstorage/trunk/relstorage/adapters/postgresql.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/adapters/postgresql.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -52,14 +52,18 @@
"""PostgreSQL adapter for RelStorage."""
implements(IRelStorageAdapter)
- def __init__(self, dsn='', keep_history=True, replica_conf=None):
+ def __init__(self, dsn='', options=None):
+ # options is a relstorage.options.Options or None
self._dsn = dsn
- self.keep_history = keep_history
- self.replica_conf = replica_conf
+ self.options = options
+ if options is not None:
+ self.keep_history = options.keep_history
+ else:
+ self.keep_history = True
self.connmanager = Psycopg2ConnectionManager(
dsn=dsn,
+ options=options,
keep_history=self.keep_history,
- replica_conf=replica_conf,
)
self.runner = ScriptRunner()
self.locker = PostgreSQLLocker(
@@ -113,9 +117,7 @@
)
def new_instance(self):
- return PostgreSQLAdapter(
- dsn=self._dsn, keep_history=self.keep_history,
- replica_conf=self.replica_conf)
+ return PostgreSQLAdapter(dsn=self._dsn, options=self.options)
def __str__(self):
parts = [self.__class__.__name__]
@@ -126,7 +128,6 @@
dsnparts = self._dsn.split()
s = ' '.join(p for p in dsnparts if not p.startswith('password'))
parts.append('dsn=%r' % s)
- parts.append('replica_conf=%r' % self.replica_conf)
return ", ".join(parts)
@@ -146,48 +147,53 @@
disconnected_exceptions = disconnected_exceptions
close_exceptions = close_exceptions
- def __init__(self, dsn, keep_history, replica_conf=None):
+ def __init__(self, dsn, options=None, keep_history=True):
self._orig_dsn = dsn
self._dsn = dsn
self.keep_history = keep_history
- self._current_replica = None
- super(Psycopg2ConnectionManager, self).__init__(
- replica_conf=replica_conf)
+ # _dsn_derived_from_replica contains the replica that
+ # was used to set self._dsn.
+ self._dsn_derived_from_replica = None
+ super(Psycopg2ConnectionManager, self).__init__(options)
def _set_dsn(self, replica):
"""Alter the DSN to use the specified replica.
The replica parameter is a string specifying either host or host:port.
"""
- if replica != self._current_replica:
+ if replica != self._dsn_derived_from_replica:
if ':' in replica:
host, port = replica.split(':')
self._dsn = self._orig_dsn + ' host=%s port=%s' % (host, port)
else:
self._dsn = self._orig_dsn + ' host=%s' % replica
- self._current_replica = replica
+ self._dsn_derived_from_replica = replica
def open(self,
isolation=psycopg2.extensions.ISOLATION_LEVEL_READ_COMMITTED):
"""Open a database connection and return (conn, cursor)."""
- if self.replicas is not None:
- self._set_dsn(self.replicas.current())
+ if self.replica_selector is not None:
+ replica = self.replica_selector.current()
+ self._set_dsn(replica)
+ else:
+ replica = None
+
while True:
try:
conn = Psycopg2Connection(self._dsn)
conn.set_isolation_level(isolation)
cursor = conn.cursor()
cursor.arraysize = 64
- conn.replica = self._current_replica
+ conn.replica = replica
return conn, cursor
except psycopg2.OperationalError, e:
- if self._current_replica:
+ if replica is not None:
log.warning("Unable to connect to replica %s: %s",
- self._current_replica, e)
+ replica, e)
else:
log.warning("Unable to connect: %s", e)
- if self.replicas is not None:
- replica = self.replicas.next()
+ if self.replica_selector is not None:
+ replica = self.replica_selector.next()
if replica is not None:
# try the new replica
self._set_dsn(replica)
Added: relstorage/trunk/relstorage/adapters/replica.py
===================================================================
--- relstorage/trunk/relstorage/adapters/replica.py (rev 0)
+++ relstorage/trunk/relstorage/adapters/replica.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -0,0 +1,111 @@
+##############################################################################
+#
+# Copyright (c) 2009 Zope Foundation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+
+from relstorage.adapters.interfaces import IReplicaSelector
+from zope.interface import implements
+import os
+import time
+
+class ReplicaSelector(object):
+ implements(IReplicaSelector)
+
+ def __init__(self, options):
+ self.replica_conf = options.replica_conf
+ self.alt_timeout = options.replica_timeout
+ self._read_config()
+ self._select(0)
+ self._iterating = False
+ self._skip_index = None
+
+ def _read_config(self):
+ self._config_modified = os.path.getmtime(self.replica_conf)
+ self._config_checked = time.time()
+ f = open(self.replica_conf, 'r')
+ try:
+ lines = f.readlines()
+ finally:
+ f.close()
+ replicas = []
+ for line in lines:
+ line = line.strip()
+ if not line or line.startswith('#'):
+ continue
+ replicas.append(line)
+ if not replicas:
+ raise IndexError(
+ "No replicas specified in %s" % self.replica_conf)
+ self._replicas = replicas
+
+ def _is_config_modified(self):
+ now = time.time()
+ if now < self._config_checked + 1:
+ # don't check the last mod time more often than once per second
+ return False
+ self._config_checked = now
+ t = os.path.getmtime(self.replica_conf)
+ return t != self._config_modified
+
+ def _select(self, index):
+ self._current_replica = self._replicas[index]
+ self._current_index = index
+ if index > 0 and self.alt_timeout:
+ self._expiration = time.time() + self.alt_timeout
+ else:
+ self._expiration = None
+
+ def current(self):
+ """Get the current replica."""
+ self._iterating = False
+ if self._is_config_modified():
+ self._read_config()
+ self._select(0)
+ elif self._expiration is not None and time.time() >= self._expiration:
+ self._select(0)
+ return self._current_replica
+
+ def next(self):
+ """Return the next replica to try.
+
+ Return None if there are no more replicas defined.
+ """
+ if self._is_config_modified():
+ # Start over even if iteration was already in progress.
+ self._read_config()
+ self._select(0)
+ self._skip_index = None
+ self._iterating = True
+ elif not self._iterating:
+ # Start iterating.
+ self._skip_index = self._current_index
+ i = 0
+ if i == self._skip_index:
+ i = 1
+ if i >= len(self._replicas):
+ # There are no more replicas to try.
+ self._select(0)
+ return None
+ self._select(i)
+ self._iterating = True
+ else:
+ # Continue iterating.
+ i = self._current_index + 1
+ if i == self._skip_index:
+ i += 1
+ if i >= len(self._replicas):
+ # There are no more replicas to try.
+ self._select(0)
+ return None
+ self._select(i)
+
+ return self._current_replica
Modified: relstorage/trunk/relstorage/adapters/tests/test_connmanager.py
===================================================================
--- relstorage/trunk/relstorage/adapters/tests/test_connmanager.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/adapters/tests/test_connmanager.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -33,10 +33,11 @@
f = tempfile.NamedTemporaryFile()
f.write("example.com:1234\n")
f.flush()
+ options = MockOptions(f.name)
from relstorage.adapters.connmanager import AbstractConnectionManager
from relstorage.adapters.interfaces import ReplicaClosedException
- cm = AbstractConnectionManager(f.name)
+ cm = AbstractConnectionManager(options)
conn = MockConnection()
conn.replica = 'example.com:1234'
@@ -55,120 +56,11 @@
cm.restart_store, conn, MockCursor())
-class ReplicaSelectorTests(unittest.TestCase):
+class MockOptions:
+ def __init__(self, fn):
+ self.replica_conf = fn
+ self.replica_timeout = 600.0
- def setUp(self):
- import tempfile
- self.f = tempfile.NamedTemporaryFile()
- self.f.write(
- "# Replicas\n\nexample.com:1234\nlocalhost:4321\n"
- "\nlocalhost:9999\n")
- self.f.flush()
-
- def tearDown(self):
- self.f.close()
-
- def test__read_config_normal(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- rs = ReplicaSelector(self.f.name)
- self.assertEqual(rs._replicas,
- ['example.com:1234', 'localhost:4321', 'localhost:9999'])
-
- def test__read_config_empty(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- self.f.seek(0)
- self.f.truncate()
- self.assertRaises(IndexError, ReplicaSelector, self.f.name)
-
- def test__is_config_modified(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- import time
- rs = ReplicaSelector(self.f.name)
- self.assertEqual(rs._is_config_modified(), False)
- # change the file
- rs._config_modified = 0
- # don't check the file yet
- rs._config_checked = time.time() + 3600
- self.assertEqual(rs._is_config_modified(), False)
- # now check the file
- rs._config_checked = 0
- self.assertEqual(rs._is_config_modified(), True)
-
- def test__select(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- rs = ReplicaSelector(self.f.name)
- rs._select(0)
- self.assertEqual(rs._current_replica, 'example.com:1234')
- self.assertEqual(rs._current_index, 0)
- self.assertEqual(rs._expiration, None)
- rs._select(1)
- self.assertEqual(rs._current_replica, 'localhost:4321')
- self.assertEqual(rs._current_index, 1)
- self.assertNotEqual(rs._expiration, None)
-
- def test_current(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- rs = ReplicaSelector(self.f.name)
- self.assertEqual(rs.current(), 'example.com:1234')
- # change the file and get the new current replica
- self.f.seek(0)
- self.f.write('localhost\nalternate\n')
- self.f.flush()
- rs._config_checked = 0
- rs._config_modified = 0
- self.assertEqual(rs.current(), 'localhost')
- # switch to the alternate
- rs._select(1)
- self.assertEqual(rs.current(), 'alternate')
- # expire the alternate
- rs._expiration = 0
- self.assertEqual(rs.current(), 'localhost')
-
- def test_next_iteration(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- rs = ReplicaSelector(self.f.name)
-
- # test forward iteration
- self.assertEqual(rs.current(), 'example.com:1234')
- self.assertEqual(rs.next(), 'localhost:4321')
- self.assertEqual(rs.next(), 'localhost:9999')
- self.assertEqual(rs.next(), None)
-
- # test iteration that skips over the replica that failed
- self.assertEqual(rs.current(), 'example.com:1234')
- self.assertEqual(rs.next(), 'localhost:4321')
- self.assertEqual(rs.current(), 'localhost:4321')
- # next() after current() indicates the last replica failed
- self.assertEqual(rs.next(), 'example.com:1234')
- self.assertEqual(rs.next(), 'localhost:9999')
- self.assertEqual(rs.next(), None)
-
- def test_next_only_one_server(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- self.f.seek(0)
- self.f.write('localhost\n')
- self.f.flush()
- self.f.truncate()
- rs = ReplicaSelector(self.f.name)
- self.assertEqual(rs.current(), 'localhost')
- self.assertEqual(rs.next(), None)
-
- def test_next_with_new_conf(self):
- from relstorage.adapters.connmanager import ReplicaSelector
- rs = ReplicaSelector(self.f.name)
- self.assertEqual(rs.current(), 'example.com:1234')
- self.assertEqual(rs.next(), 'localhost:4321')
- # interrupt the iteration by changing the replica conf file
- self.f.seek(0)
- self.f.write('example.com:9999\n')
- self.f.flush()
- self.f.truncate()
- rs._config_checked = 0
- rs._config_modified = 0
- self.assertEqual(rs.next(), 'example.com:9999')
- self.assertEqual(rs.next(), None)
-
-
class MockConnection:
def rollback(self):
self.rolled_back = True
@@ -183,9 +75,5 @@
def test_suite():
suite = unittest.TestSuite()
- for klass in [
- AbstractConnectionManagerTests,
- ReplicaSelectorTests,
- ]:
- suite.addTest(unittest.makeSuite(klass))
+ suite.addTest(unittest.makeSuite(AbstractConnectionManagerTests))
return suite
Added: relstorage/trunk/relstorage/adapters/tests/test_replica.py
===================================================================
--- relstorage/trunk/relstorage/adapters/tests/test_replica.py (rev 0)
+++ relstorage/trunk/relstorage/adapters/tests/test_replica.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -0,0 +1,140 @@
+##############################################################################
+#
+# Copyright (c) 2009 Zope Foundation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+
+import unittest
+
+class ReplicaSelectorTests(unittest.TestCase):
+
+ def setUp(self):
+ import tempfile
+ self.f = tempfile.NamedTemporaryFile()
+ self.f.write(
+ "# Replicas\n\nexample.com:1234\nlocalhost:4321\n"
+ "\nlocalhost:9999\n")
+ self.f.flush()
+ self.options = MockOptions(self.f.name)
+
+ def tearDown(self):
+ self.f.close()
+
+ def test__read_config_normal(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ rs = ReplicaSelector(self.options)
+ self.assertEqual(rs._replicas,
+ ['example.com:1234', 'localhost:4321', 'localhost:9999'])
+
+ def test__read_config_empty(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ self.f.seek(0)
+ self.f.truncate()
+ self.assertRaises(IndexError, ReplicaSelector, self.options)
+
+ def test__is_config_modified(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ import time
+ rs = ReplicaSelector(self.options)
+ self.assertEqual(rs._is_config_modified(), False)
+ # change the file
+ rs._config_modified = 0
+ # don't check the file yet
+ rs._config_checked = time.time() + 3600
+ self.assertEqual(rs._is_config_modified(), False)
+ # now check the file
+ rs._config_checked = 0
+ self.assertEqual(rs._is_config_modified(), True)
+
+ def test__select(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ rs = ReplicaSelector(self.options)
+ rs._select(0)
+ self.assertEqual(rs._current_replica, 'example.com:1234')
+ self.assertEqual(rs._current_index, 0)
+ self.assertEqual(rs._expiration, None)
+ rs._select(1)
+ self.assertEqual(rs._current_replica, 'localhost:4321')
+ self.assertEqual(rs._current_index, 1)
+ self.assertNotEqual(rs._expiration, None)
+
+ def test_current(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ rs = ReplicaSelector(self.options)
+ self.assertEqual(rs.current(), 'example.com:1234')
+ # change the file and get the new current replica
+ self.f.seek(0)
+ self.f.write('localhost\nalternate\n')
+ self.f.flush()
+ rs._config_checked = 0
+ rs._config_modified = 0
+ self.assertEqual(rs.current(), 'localhost')
+ # switch to the alternate
+ rs._select(1)
+ self.assertEqual(rs.current(), 'alternate')
+ # expire the alternate
+ rs._expiration = 0
+ self.assertEqual(rs.current(), 'localhost')
+
+ def test_next_iteration(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ rs = ReplicaSelector(self.options)
+
+ # test forward iteration
+ self.assertEqual(rs.current(), 'example.com:1234')
+ self.assertEqual(rs.next(), 'localhost:4321')
+ self.assertEqual(rs.next(), 'localhost:9999')
+ self.assertEqual(rs.next(), None)
+
+ # test iteration that skips over the replica that failed
+ self.assertEqual(rs.current(), 'example.com:1234')
+ self.assertEqual(rs.next(), 'localhost:4321')
+ self.assertEqual(rs.current(), 'localhost:4321')
+ # next() after current() indicates the last replica failed
+ self.assertEqual(rs.next(), 'example.com:1234')
+ self.assertEqual(rs.next(), 'localhost:9999')
+ self.assertEqual(rs.next(), None)
+
+ def test_next_only_one_server(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ self.f.seek(0)
+ self.f.write('localhost\n')
+ self.f.flush()
+ self.f.truncate()
+ rs = ReplicaSelector(self.options)
+ self.assertEqual(rs.current(), 'localhost')
+ self.assertEqual(rs.next(), None)
+
+ def test_next_with_new_conf(self):
+ from relstorage.adapters.replica import ReplicaSelector
+ rs = ReplicaSelector(self.options)
+ self.assertEqual(rs.current(), 'example.com:1234')
+ self.assertEqual(rs.next(), 'localhost:4321')
+ # interrupt the iteration by changing the replica conf file
+ self.f.seek(0)
+ self.f.write('example.com:9999\n')
+ self.f.flush()
+ self.f.truncate()
+ rs._config_checked = 0
+ rs._config_modified = 0
+ self.assertEqual(rs.next(), 'example.com:9999')
+ self.assertEqual(rs.next(), None)
+
+
+class MockOptions:
+ def __init__(self, fn):
+ self.replica_conf = fn
+ self.replica_timeout = 600.0
+
+def test_suite():
+ suite = unittest.TestSuite()
+ suite.addTest(unittest.makeSuite(ReplicaSelectorTests))
+ return suite
Modified: relstorage/trunk/relstorage/component.xml
===================================================================
--- relstorage/trunk/relstorage/component.xml 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/component.xml 2009-10-03 21:54:16 UTC (rev 104768)
@@ -11,12 +11,6 @@
datatype=".RelStorageFactory">
<section type="relstorage.adapter" name="*" attribute="adapter"/>
<key name="name" datatype="string" required="no"/>
- <key name="create" datatype="boolean" default="true">
- <description>
- Flag that indicates whether the storage should be initialized if
- it does not already exist.
- </description>
- </key>
<key name="read-only" datatype="boolean" default="false">
<description>
If true, only reads may be executed against the storage. Note
@@ -32,6 +26,50 @@
is provided, then no blob support will be provided.
</description>
</key>
+ <key name="keep-history" datatype="boolean" default="true">
+ <description>
+ If this parameter is set to true (the default), the adapter
+ will create and use a history-preserving database schema,
+ similar to FileStorage. A history-preserving schema supports
+ ZODB-level undo, but grows more quickly and requires extensive
+ packing on a regular basis.
+
+ If this parameter is set to false, the adapter will create and
+ use a history-free database schema. Undo will not be supported,
+ but the database will not grow as quickly. The database will
+ still require regular garbage collection (which is accessible
+ through the database pack mechanism.)
+
+ This parameter must not change once the database schema has
+ been installed. If you want to convert between a
+ history-preserving and a history-free database, use the
+ zodbconvert utility to copy to a new database.
+ </description>
+ </key>
+ <key name="replica-conf" datatype="string" required="no">
+ <description>
+ If this parameter is provided, it specifies a text file that
+ contains a list of database replicas this adapter can choose
+ from. For MySQL and PostgreSQL, put in the replica file a list
+ of ``host:port`` or ``host`` values, one per line. For Oracle,
+ put in a list of DSN values. Blank lines and lines starting
+ with ``#`` are ignored.
+
+ The adapter prefers the first replica specified in the file. If
+ the first is not available, the adapter automatically tries the
+ rest of the replicas, in order. If the file changes, the
+ adapter will drop existing SQL database connections and make
+ new connections when ZODB starts a new transaction.
+ </description>
+ </key>
+ <key name="replica-timeout" datatype="float" default="600.0">
+ <description>
+ If this parameter has a nonzero value, when the adapter selects
+ a replica other than the primary replica, the adapter will
+ try to revert to the primary replica after the specified
+ timeout (in seconds). The default is 600, meaning 10 minutes.
+ </description>
+ </key>
<key name="poll-interval" datatype="float" required="no">
<description>
Defer polling the database for the specified maximum time interval,
@@ -135,47 +173,8 @@
</key>
</sectiontype>
- <sectiontype name="relstorage.adapter.common">
- <key name="keep-history" datatype="boolean" default="true">
- <description>
- If this parameter is set to true (the default), the adapter
- will create and use a history-preserving database schema,
- similar to FileStorage. A history-preserving schema supports
- ZODB-level undo, but grows more quickly and requires extensive
- packing on a regular basis.
-
- If this parameter is set to false, the adapter will create and
- use a history-free database schema. Undo will not be supported,
- but the database will not grow as quickly. The database will
- still require regular garbage collection (which is accessible
- through the database pack mechanism.)
-
- This parameter must not change once the database schema has
- been installed. If you want to convert between a
- history-preserving and a history-free database, use the
- zodbconvert utility to copy to a new database.
- </description>
- </key>
- <key name="replica-conf" datatype="string" required="no">
- <description>
- If this parameter is provided, it specifies a text file that
- contains a list of database replicas this adapter can choose
- from. For MySQL and PostgreSQL, put in the replica file a list
- of ``host:port`` or ``host`` values, one per line. For Oracle,
- put in a list of DSN values. Blank lines and lines starting
- with ``#`` are ignored.
-
- The adapter prefers the first replica specified in the file. If
- the first is not available, the adapter automatically tries the
- rest of the replicas, in order. If the file changes, the
- adapter will drop existing SQL database connections and make
- new connections when ZODB starts a new transaction.
- </description>
- </key>
- </sectiontype>
-
<sectiontype name="postgresql" implements="relstorage.adapter"
- datatype=".PostgreSQLAdapterFactory" extends="relstorage.adapter.common">
+ datatype=".PostgreSQLAdapterFactory">
<key name="dsn" datatype="string" required="no" default="">
<description>
The PostgreSQL data source name. For example:
@@ -190,7 +189,7 @@
</sectiontype>
<sectiontype name="oracle" implements="relstorage.adapter"
- datatype=".OracleAdapterFactory" extends="relstorage.adapter.common">
+ datatype=".OracleAdapterFactory">
<key name="user" datatype="string" required="yes">
<description>
The Oracle account name
@@ -210,7 +209,7 @@
</sectiontype>
<sectiontype name="mysql" implements="relstorage.adapter"
- datatype=".MySQLAdapterFactory" extends="relstorage.adapter.common">
+ datatype=".MySQLAdapterFactory">
<key name="host" datatype="string" required="no">
<description>
host to connect
Modified: relstorage/trunk/relstorage/config.py
===================================================================
--- relstorage/trunk/relstorage/config.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/config.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -15,53 +15,52 @@
from ZODB.config import BaseConfig
-from relstorage.storage import RelStorage, Options
+from relstorage.options import Options
+from relstorage.storage import RelStorage
+from relstorage.adapters.replica import ReplicaSelector
class RelStorageFactory(BaseConfig):
"""Open a storage configured via ZConfig"""
def open(self):
config = self.config
- adapter = config.adapter.create()
options = Options()
for key in options.__dict__.keys():
value = getattr(config, key, None)
if value is not None:
setattr(options, key, value)
- return RelStorage(adapter, name=config.name, create=config.create,
- read_only=config.read_only, options=options)
+ adapter = config.adapter.create(options)
+ return RelStorage(adapter, name=config.name, options=options)
class PostgreSQLAdapterFactory(BaseConfig):
- def create(self):
+ def create(self, options):
from adapters.postgresql import PostgreSQLAdapter
return PostgreSQLAdapter(
dsn=self.config.dsn,
- keep_history=self.config.keep_history,
- replica_conf=self.config.replica_conf,
+ options=options,
)
class OracleAdapterFactory(BaseConfig):
- def create(self):
+ def create(self, options):
from adapters.oracle import OracleAdapter
config = self.config
return OracleAdapter(
user=config.user,
password=config.password,
dsn=config.dsn,
- keep_history=config.keep_history,
- replica_conf=config.replica_conf,
+ options=options,
)
class MySQLAdapterFactory(BaseConfig):
- def create(self):
+ def create(self, options):
from adapters.mysql import MySQLAdapter
- options = {}
+ params = {}
for key in self.config.getSectionAttributes():
value = getattr(self.config, key)
if value is not None:
- options[key] = value
- return MySQLAdapter(**options)
+ params[key] = value
+ return MySQLAdapter(options=options, **params)
Added: relstorage/trunk/relstorage/options.py
===================================================================
--- relstorage/trunk/relstorage/options.py (rev 0)
+++ relstorage/trunk/relstorage/options.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -0,0 +1,46 @@
+##############################################################################
+#
+# Copyright (c) 2008 Zope Foundation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+
+class Options(object):
+ """Options for configuring and tuning RelStorage.
+
+ These parameters can be provided as keyword options in the RelStorage
+ constructor. For example:
+
+ storage = RelStorage(adapter, pack_gc=True, pack_dry_run=True)
+
+ Alternatively, the RelStorage constructor accepts an options
+ parameter, which should be an Options instance.
+ """
+ def __init__(self, **kwoptions):
+ self.name = None
+ self.read_only = False
+ self.blob_dir = None
+ self.keep_history = True
+ self.replica_conf = None
+ self.replica_timeout = 600.0
+ self.poll_interval = 0
+ self.pack_gc = True
+ self.pack_dry_run = False
+ self.pack_batch_timeout = 5.0
+ self.pack_duty_cycle = 0.5
+ self.pack_max_delay = 20.0
+ self.cache_servers = () # ['127.0.0.1:11211']
+ self.cache_module_name = 'memcache'
+
+ for key, value in kwoptions.iteritems():
+ if key in self.__dict__:
+ setattr(self, key, value)
+ else:
+ raise TypeError("Unknown parameter: %s" % key)
Modified: relstorage/trunk/relstorage/storage.py
===================================================================
--- relstorage/trunk/relstorage/storage.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/storage.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -17,6 +17,7 @@
"""
from persistent.TimeStamp import TimeStamp
+from relstorage.options import Options
from relstorage.util import is_blob_record
from ZODB.BaseStorage import BaseStorage
from ZODB.BaseStorage import DataRecord
@@ -75,24 +76,23 @@
implements(*_relstorage_interfaces)
def __init__(self, adapter, name=None, create=True,
- read_only=False, options=None, **kwoptions):
- if name is None:
- name = 'RelStorage: %s' % adapter
-
+ options=None, **kwoptions):
self._adapter = adapter
- self._name = name
- self._is_read_only = read_only
+
if options is None:
- options = Options()
- for key, value in kwoptions.iteritems():
- if key in options.__dict__:
- setattr(options, key, value)
- else:
- raise TypeError("Unknown parameter: %s" % key)
+ options = Options(**kwoptions)
elif kwoptions:
raise TypeError("The RelStorage constructor accepts either "
"an options parameter or keyword arguments, not both")
self._options = options
+
+ if not name:
+ name = options.name
+ if not name:
+ name = 'RelStorage: %s' % adapter
+ self._name = name
+
+ self._is_read_only = options.read_only
self._cache_client = None
if create:
@@ -285,8 +285,7 @@
"""
adapter = self._adapter.new_instance()
other = RelStorage(adapter=adapter, name=self._name,
- create=False, read_only=self._is_read_only,
- options=self._options)
+ create=False, options=self._options)
self._instances.append(weakref.ref(other))
return other
@@ -1422,26 +1421,3 @@
self.data = str(data)
else:
self.data = None
-
-
-class Options:
- """Options for tuning RelStorage.
-
- These parameters can be provided as keyword options in the RelStorage
- constructor. For example:
-
- storage = RelStorage(adapter, pack_gc=True, pack_dry_run=True)
-
- Alternatively, the RelStorage constructor accepts an options
- parameter, which should be an Options instance.
- """
- def __init__(self):
- self.blob_dir = None
- self.poll_interval = 0
- self.pack_gc = True
- self.pack_dry_run = False
- self.pack_batch_timeout = 5.0
- self.pack_duty_cycle = 0.5
- self.pack_max_delay = 20.0
- self.cache_servers = () # ['127.0.0.1:11211']
- self.cache_module_name = 'memcache'
Modified: relstorage/trunk/relstorage/tests/testmysql.py
===================================================================
--- relstorage/trunk/relstorage/tests/testmysql.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/tests/testmysql.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -13,6 +13,7 @@
##############################################################################
"""Tests of relstorage.adapters.mysql"""
+from relstorage.options import Options
from relstorage.tests.hftestbase import HistoryFreeFromFileStorage
from relstorage.tests.hftestbase import HistoryFreeRelStorageTests
from relstorage.tests.hftestbase import HistoryFreeToFileStorage
@@ -31,7 +32,7 @@
else:
db = 'relstoragetest_hf'
return MySQLAdapter(
- keep_history=self.keep_history,
+ options=Options(keep_history=self.keep_history),
db=db,
user='relstoragetest',
passwd='relstoragetest',
@@ -55,19 +56,21 @@
%%import relstorage
<zodb main>
<relstorage>
+ name xyz
+ read-only false
+ keep-history %s
+ replica-conf %s
<mysql>
db %s
user relstoragetest
passwd relstoragetest
- keep-history %s
- replica-conf %s
</mysql>
</relstorage>
</zodb>
""" % (
- dbname,
self.keep_history and 'true' or 'false',
replica_conf.name,
+ dbname,
)
schema_xml = """
@@ -84,6 +87,8 @@
db = config.database.open()
try:
storage = db.storage
+ self.assertEqual(storage._is_read_only, False)
+ self.assertEqual(storage._name, "xyz")
adapter = storage._adapter
from relstorage.adapters.mysql import MySQLAdapter
self.assert_(isinstance(adapter, MySQLAdapter))
@@ -91,9 +96,11 @@
'passwd': 'relstoragetest',
'db': dbname,
'user': 'relstoragetest',
- 'replica_conf': replica_conf.name,
})
self.assertEqual(adapter.keep_history, self.keep_history)
+ self.assertEqual(
+ adapter.connmanager.replica_selector.replica_conf,
+ replica_conf.name)
finally:
db.close()
finally:
@@ -154,7 +161,7 @@
if not keep_history:
db += '_hf'
adapter = MySQLAdapter(
- keep_history=keep_history,
+ options=Options(keep_history=keep_history),
db=db,
user='relstoragetest',
passwd='relstoragetest',
Modified: relstorage/trunk/relstorage/tests/testoracle.py
===================================================================
--- relstorage/trunk/relstorage/tests/testoracle.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/tests/testoracle.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -13,6 +13,7 @@
##############################################################################
"""Tests of relstorage.adapters.oracle"""
+from relstorage.options import Options
from relstorage.tests.hftestbase import HistoryFreeFromFileStorage
from relstorage.tests.hftestbase import HistoryFreeRelStorageTests
from relstorage.tests.hftestbase import HistoryFreeToFileStorage
@@ -32,10 +33,10 @@
else:
db = 'relstoragetest_hf'
return OracleAdapter(
- keep_history=self.keep_history,
user=db,
password='relstoragetest',
dsn=dsn,
+ options=Options(keep_history=self.keep_history),
)
@@ -57,20 +58,22 @@
%%import relstorage
<zodb main>
<relstorage>
+ name xyz
+ read-only false
+ keep-history %s
+ replica-conf %s
<oracle>
user %s
password relstoragetest
dsn %s
- keep-history %s
- replica-conf %s
</oracle>
</relstorage>
</zodb>
""" % (
+ self.keep_history and 'true' or 'false',
+ replica_conf.name,
dbname,
dsn,
- self.keep_history and 'true' or 'false',
- replica_conf.name,
)
schema_xml = """
@@ -87,6 +90,8 @@
db = config.database.open()
try:
storage = db.storage
+ self.assertEqual(storage._is_read_only, False)
+ self.assertEqual(storage._name, "xyz")
adapter = storage._adapter
from relstorage.adapters.oracle import OracleAdapter
self.assert_(isinstance(adapter, OracleAdapter))
@@ -95,7 +100,9 @@
self.assertEqual(adapter._dsn, dsn)
self.assertEqual(adapter._twophase, False)
self.assertEqual(adapter.keep_history, self.keep_history)
- self.assertEqual(adapter.replica_conf, replica_conf.name)
+ self.assertEqual(
+ adapter.connmanager.replica_selector.replica_conf,
+ replica_conf.name)
finally:
db.close()
finally:
@@ -157,10 +164,10 @@
if not keep_history:
db += '_hf'
adapter = OracleAdapter(
- keep_history=keep_history,
user=db,
password='relstoragetest',
dsn=dsn,
+ options=Options(keep_history=keep_history),
)
storage = RelStorage(adapter, name=name, create=True,
blob_dir=os.path.abspath(blob_dir))
Modified: relstorage/trunk/relstorage/tests/testpostgresql.py
===================================================================
--- relstorage/trunk/relstorage/tests/testpostgresql.py 2009-10-03 19:39:22 UTC (rev 104767)
+++ relstorage/trunk/relstorage/tests/testpostgresql.py 2009-10-03 21:54:16 UTC (rev 104768)
@@ -13,6 +13,7 @@
##############################################################################
"""Tests of relstorage.adapters.postgresql"""
+from relstorage.options import Options
from relstorage.tests.hftestbase import HistoryFreeFromFileStorage
from relstorage.tests.hftestbase import HistoryFreeRelStorageTests
from relstorage.tests.hftestbase import HistoryFreeToFileStorage
@@ -31,8 +32,8 @@
else:
db = 'relstoragetest_hf'
return PostgreSQLAdapter(
- keep_history=self.keep_history,
- dsn='dbname=%s user=relstoragetest password=relstoragetest' % db
+ dsn='dbname=%s user=relstoragetest password=relstoragetest' % db,
+ options=Options(keep_history=self.keep_history),
)
@@ -56,17 +57,19 @@
%%import relstorage
<zodb main>
<relstorage>
+ name xyz
+ read-only false
+ keep-history %s
+ replica-conf %s
<postgresql>
dsn %s
- keep-history %s
- replica-conf %s
</postgresql>
</relstorage>
</zodb>
""" % (
- dsn,
self.keep_history and 'true' or 'false',
replica_conf.name,
+ dsn,
)
schema_xml = """
@@ -83,12 +86,16 @@
db = config.database.open()
try:
storage = db.storage
+ self.assertEqual(storage._is_read_only, False)
+ self.assertEqual(storage._name, "xyz")
adapter = storage._adapter
from relstorage.adapters.postgresql import PostgreSQLAdapter
self.assert_(isinstance(adapter, PostgreSQLAdapter))
self.assertEqual(adapter._dsn, dsn)
self.assertEqual(adapter.keep_history, self.keep_history)
- self.assertEqual(adapter.replica_conf, replica_conf.name)
+ self.assertEqual(
+ adapter.connmanager.replica_selector.replica_conf,
+ replica_conf.name)
finally:
db.close()
finally:
@@ -152,7 +159,7 @@
dsn = ('dbname=%s user=relstoragetest '
'password=relstoragetest' % db)
adapter = PostgreSQLAdapter(
- keep_history=keep_history, dsn=dsn)
+ dsn=dsn, options=Options(keep_history=keep_history))
storage = RelStorage(adapter, name=name, create=True,
blob_dir=os.path.abspath(blob_dir))
storage.zap_all()
More information about the checkins
mailing list