[Checkins] SVN: mongopersist/trunk/ - Bug: When a transaction was aborted, the states of all *loaded* objects were
Stephen Richter
cvs-admin at zope.org
Thu Mar 29 14:41:41 UTC 2012
Log message for revision 124793:
- Bug: When a transaction was aborted, the states of all *loaded* objects were
reset. Now, only *modified* object states are reset. This should drastically
lower problems (by the ratio of read over modified objects) due to lack of
full MVCC.
Changed:
U mongopersist/trunk/CHANGES.txt
U mongopersist/trunk/setup.py
U mongopersist/trunk/src/mongopersist/datamanager.py
U mongopersist/trunk/src/mongopersist/tests/test_datamanager.py
-=-
Modified: mongopersist/trunk/CHANGES.txt
===================================================================
--- mongopersist/trunk/CHANGES.txt 2012-03-29 13:49:43 UTC (rev 124792)
+++ mongopersist/trunk/CHANGES.txt 2012-03-29 14:41:37 UTC (rev 124793)
@@ -2,22 +2,25 @@
CHANGES
=======
-0.6.2 (unreleased)
+0.7.0 (2012-03-??)
------------------
-- Nothing changed yet.
+- Bug: When a transaction was aborted, the states of all *loaded* objects were
+ reset. Now, only *modified* object states are reset. This should drastically
+ lower problems (by the ratio of read over modified objects) due to lack of
+ full MVCC.
0.6.1 (2012-03-28)
------------------
-- Added quite detailed debug logging around collection methods
+- Feature: Added quite detailed debug logging around collection methods
0.6.0 (2012-03-12)
------------------
-- Switched to optimisitc data dumping, which approaches transactions by
- dumping early and as the data comes. All changes are undone when the
+- Feature: Switched to optimisitc data dumping, which approaches transactions
+ by dumping early and as the data comes. All changes are undone when the
transaction fails/aborts. See ``optimistic-data-dumping.txt`` for
details. Here are some of the new features:
@@ -53,36 +56,36 @@
coll_name)`` and ``DataManager.get_collection_from_object(obj)``
allows one to quickly get a wrapped collection.
-- Renamed ``processSpec()`` to ``process_spec()`` to adhere to package nameing
- convention.
+- Feature: Renamed ``processSpec()`` to ``process_spec()`` to adhere to
+ package nameing convention.
-- Created a ``ProcessSpecDecorator`` that is used in the ``CollectionWrapper``
- class to process the specs of the ``find()``, ``find_one()`` and
- ``find_and_modify()`` collection methods.
+- Feature: Created a ``ProcessSpecDecorator`` that is used in the
+ ``CollectionWrapper`` class to process the specs of the ``find()``,
+ ``find_one()`` and ``find_and_modify()`` collection methods.
-- The ``MongoContainer`` class now removes objects from the database upon
- container removal is ``_m_remove_documents`` is ``True``. The default is
- ``True``.
+- Feature: The ``MongoContainer`` class now removes objects from the database
+ upon container removal is ``_m_remove_documents`` is ``True``. The default
+ is ``True``.
-- When adding an item to ``MongoContainer`` and the key is ``None``, then the
- OID is chosen as the key. Ids are perfect key, because they are guaranteed
- to be unique within the collection.
+- Feature: When adding an item to ``MongoContainer`` and the key is ``None``,
+ then the OID is chosen as the key. Ids are perfect keys, because they are
+ guaranteed to be unique within the collection.
-- Since people did not like the setitem with ``None`` key implementation, I
- also added the ``MongoContainer.add(value, key=None)`` method, which makes
- specifying the key optional. The default implementation is to use the OID,
- if the key is ``None``.
+- Feature: Since people did not like the setitem with ``None`` key
+ implementation, I also added the ``MongoContainer.add(value, key=None)``
+ method, which makes specifying the key optional. The default implementation
+ is to use the OID, if the key is ``None``.
-- Removed ``fields`` argument from the ``MongoContainer.find(...)`` and
- ``MongoContainer.find_one(...)`` methods, since it was not used.
+- Feature: Removed ``fields`` argument from the ``MongoContainer.find(...)``
+ and ``MongoContainer.find_one(...)`` methods, since it was not used.
-- If a container has N items, it took N+1 queries to load the list of items
- completely. This was due to one query returning all DBRefs and then using
- one query to load the state for each. Now, the first query loads all full
- states and uses an extension to ``DataManager.setstate(obj, doc=None)`` to
- load the state of the object with the previously queried data.
+- Feature: If a container has N items, it took N+1 queries to load the list of
+ items completely. This was due to one query returning all DBRefs and then
+ using one query to load the state for each. Now, the first query loads all
+ full states and uses an extension to ``DataManager.setstate(obj, doc=None)``
+ to load the state of the object with the previously queried data.
-- Changed ``MongoContainer.get_collection()`` to return a
+- Feature: Changed ``MongoContainer.get_collection()`` to return a
``CollectionWrapper`` instance.
Modified: mongopersist/trunk/setup.py
===================================================================
--- mongopersist/trunk/setup.py 2012-03-29 13:49:43 UTC (rev 124792)
+++ mongopersist/trunk/setup.py 2012-03-29 14:41:37 UTC (rev 124793)
@@ -9,7 +9,7 @@
setup (
name='mongopersist',
- version='0.6.2.dev0',
+ version='0.7.0.dev0',
author = "Stephan Richter",
author_email = "stephan.richter at gmail.com",
description = "Mongo Persistence Backend",
Modified: mongopersist/trunk/src/mongopersist/datamanager.py
===================================================================
--- mongopersist/trunk/src/mongopersist/datamanager.py 2012-03-29 13:49:43 UTC (rev 124792)
+++ mongopersist/trunk/src/mongopersist/datamanager.py 2012-03-29 14:41:37 UTC (rev 124793)
@@ -186,6 +186,7 @@
self._registered_objects = []
self._loaded_objects = []
self._inserted_objects = []
+ self._modified_objects = []
self._removed_objects = []
self._original_states = {}
self._needs_to_join = True
@@ -325,6 +326,8 @@
if obj is not None and obj not in self._registered_objects:
self._registered_objects.append(obj)
+ if obj is not None and obj not in self._modified_objects:
+ self._modified_objects.append(obj)
def abort(self, transaction):
# Aborting the transaction requires three steps:
@@ -338,8 +341,18 @@
coll.insert(self._original_states[obj._p_oid])
del self._original_states[obj._p_oid]
# 3. Reset any changed states.
- for db_ref, state in self._original_states.items():
+ for obj in self._modified_objects:
+ db_ref = obj._p_oid
+ state = self._original_states.get(db_ref)
+ if state is None:
+ # This should not happen in a fully running environment, but
+ # the tests abort transactions often without having loaded
+ # objects through proper channels.
+ continue
coll = self.get_collection(db_ref.database, db_ref.collection)
+ # XXX: There should be a check here whether the state has been
+ # modified in the mean time by another transaction. Then a policy
+ # needs to decide what to do.
coll.update({'_id': db_ref.id}, state, True)
self.reset()
Modified: mongopersist/trunk/src/mongopersist/tests/test_datamanager.py
===================================================================
--- mongopersist/trunk/src/mongopersist/tests/test_datamanager.py 2012-03-29 13:49:43 UTC (rev 124792)
+++ mongopersist/trunk/src/mongopersist/tests/test_datamanager.py 2012-03-29 14:41:37 UTC (rev 124793)
@@ -14,7 +14,6 @@
"""Mongo Tests"""
import doctest
import persistent
-import pprint
import transaction
from pymongo import dbref, objectid
@@ -508,72 +507,78 @@
{u'_id': ObjectId('4f5c114f37a08e2cac000001'), u'name': u'two'})
"""
-def doctest_MongoDataManager_commit():
- r"""MongoDataManager: commit()
+def doctest_MongoDataManager_abort_modified_only():
+ r"""MongoDataManager: abort(): Only reset changed objects.
- Contrary to what the name suggests, this is the commit called during the
- first phase of a two-phase commit. Thus, for all practically purposes,
- this method merely checks whether the commit would potentially fail.
+ We want to make sure that we only reset modified objects, not all objects
+ that have been loaded. The ratio from reads to writes is very high, so
+ unexpected behavior with other transactions is decreased by that ratio.
- This means, if conflict detection is disabled, this method does nothing.
+ First let's create an initial state:
- >>> dm.detect_conflicts
- False
- >>> dm.commit(transaction.get())
+ >>> dm.reset()
+ >>> foo1_ref = dm.insert(Foo('one'))
+ >>> foo2_ref = dm.insert(Foo('two'))
+ >>> foo3_ref = dm.insert(Foo('three'))
+ >>> dm.reset()
- Let's now turn on conflict detection:
+ >>> coll = dm._get_collection_from_object(Foo())
+ >>> tuple(coll.find({}))
+ ({u'_id': ObjectId('4f5c114f37a08e2cac000000'), u'name': u'one'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000001'), u'name': u'two'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000002'), u'name': u'three'})
- >>> dm.detect_conflicts = True
+ 1. Transaction A loads all objects:
- For new objects (not having an oid), it always passes:
+ >>> foo1_A = dm.load(foo1_ref)
+ >>> foo1_A.name
+ u'one'
+ >>> foo2_A = dm.load(foo2_ref)
+ >>> foo2_A.name
+ u'two'
+ >>> foo3_A = dm.load(foo3_ref)
+ >>> foo3_A.name
+ u'three'
- >>> dm.reset()
- >>> dm._registered_objects = [Foo()]
- >>> dm.commit(transaction.get())
+ >>> sorted([ref.id for ref in dm._original_states.keys()])
+ [ObjectId('4f746d0b37a08e1013000000'),
+ ObjectId('4f746d0b37a08e1013000001'),
+ ObjectId('4f746d0b37a08e1013000002')]
- If the object has an oid, but is not found in the DB, we also just pass,
- because the object will be inserted.
+ 2. Transaction B comes along and modifies Foo 3's data and commits:
- >>> foo = Foo()
- >>> foo._p_oid = dbref.DBRef(
- ... 'mongopersist.tests.test_datamanager.Foo',
- ... objectid.ObjectId('4eb2eb7437a08e0156000000'),
- ... 'mongopersist_test')
+ >>> dm_B = datamanager.MongoDataManager(
+ ... conn, default_database=DBNAME, root_database=DBNAME)
- >>> dm.reset()
- >>> dm._registered_objects = [foo]
- >>> dm.commit(transaction.get())
+ >>> foo3_B = dm_B.load(foo3_ref)
+ >>> foo3_B.name = '3'
+ >>> dm_B.tpc_finish(None)
- Let's now store an object and make sure it does not conflict:
+ >>> tuple(coll.find({}))
+ ({u'_id': ObjectId('4f5c114f37a08e2cac000000'), u'name': u'one'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000001'), u'name': u'two'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000002'), u'name': u'3'})
- >>> foo = Foo()
- >>> ref = dm.dump(foo)
- >>> ref
- DBRef('mongopersist.tests.test_datamanager.Foo',
- ObjectId('4eb3468037a08e1b74000000'),
- 'mongopersist_test')
+ 3. Transaction A modifies Foo 1 and the data is flushed:
- >>> dm.reset()
- >>> dm._registered_objects = [foo]
- >>> dm.commit(transaction.get())
+ >>> foo1_A.name = '1'
+ >>> dm.flush()
- Next, let's cause a conflict byt simulating a conflicting transaction:
+ >>> tuple(coll.find({}))
+ ({u'_id': ObjectId('4f5c114f37a08e2cac000000'), u'name': u'1'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000001'), u'name': u'two'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000002'), u'name': u'3'})
- >>> dm.reset()
- >>> foo2 = dm.load(ref)
- >>> foo2.name = 'foo2'
- >>> transaction.commit()
+ 4. If transcation A is later aborted, only objects modified within the
+ transaction get reset to their original state (and not all loaded ones:
- >>> dm.reset()
- >>> dm._registered_objects = [foo]
- >>> dm.commit(transaction.get())
- Traceback (most recent call last):
- ...
- ConflictError: database conflict error
- (oid DBRef('mongopersist.tests.test_datamanager.Foo',
- ObjectId('4eb3499637a08e1c5a000000'),
- 'mongopersist_test'),
- class Foo, start serial 1, current serial 2)
+ >>> dm.abort(None)
+
+ >>> tuple(coll.find({}))
+ ({u'_id': ObjectId('4f5c114f37a08e2cac000000'), u'name': u'one'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000001'), u'name': u'two'},
+ {u'_id': ObjectId('4f5c114f37a08e2cac000002'), u'name': u'3'})
+
"""
def doctest_MongoDataManager_tpc_begin():
More information about the checkins
mailing list