[Checkins] [zopefoundation/ZODB] 9af7c7: Fix test compatibility with persistent 4.6; test C...
Alessandro Pisa
noreply at github.com
Tue Sep 22 15:20:17 CEST 2020
Branch: refs/heads/fix_UnboundLocalError
Home: https://github.com/zopefoundation/ZODB
Commit: 9af7c7f3e0e17bde1a9e3e70427961711e494136
https://github.com/zopefoundation/ZODB/commit/9af7c7f3e0e17bde1a9e3e70427961711e494136
Author: Jason Madden <jamadden at gmail.com>
Date: 2020-03-05 (Thu, 05 Mar 2020)
Changed paths:
M .travis.yml
M src/ZODB/tests/testCache.py
M src/ZODB/tests/testConnectionSavepoint.py
M src/ZODB/tests/testThreadedShutdown.py
M tox.ini
Log Message:
-----------
Fix test compatibility with persistent 4.6; test CPython with PURE_PYTHON.
Commit: 6601d63420b71bddd36a1b4b7d5c8fd697f6b318
https://github.com/zopefoundation/ZODB/commit/6601d63420b71bddd36a1b4b7d5c8fd697f6b318
Author: Jason Madden <jamadden at gmail.com>
Date: 2020-03-05 (Thu, 05 Mar 2020)
Changed paths:
M .travis.yml
M src/ZODB/tests/testCache.py
M src/ZODB/tests/testConnectionSavepoint.py
M src/ZODB/tests/testThreadedShutdown.py
M tox.ini
Log Message:
-----------
Merge pull request #296 from zopefoundation/pure-python-tests
Fix test compatibility with persistent 4.6; test CPython with PURE_PYTHON
Commit: 29225f21135214b3090d94adfa06c8c065e0a6ea
https://github.com/zopefoundation/ZODB/commit/29225f21135214b3090d94adfa06c8c065e0a6ea
Author: Jason Madden <jamadden at gmail.com>
Date: 2020-03-05 (Thu, 05 Mar 2020)
Changed paths:
M appveyor.yml
Log Message:
-----------
Simplify appveyor.yml.
Fixes issues with zc.buildout and version pins (https://github.com/zopefoundation/ZODB/pull/295#issuecomment-593805066)
Commit: 22055bdeb1e18abdc0e8ee48c6fcd46bfdbcde08
https://github.com/zopefoundation/ZODB/commit/22055bdeb1e18abdc0e8ee48c6fcd46bfdbcde08
Author: Jason Madden <jamadden at gmail.com>
Date: 2020-03-05 (Thu, 05 Mar 2020)
Changed paths:
M appveyor.yml
Log Message:
-----------
Evidently the build_script is required.
Commit: 0e343a8cdb3ca5a42e81a295ae2c4fb81533ef23
https://github.com/zopefoundation/ZODB/commit/0e343a8cdb3ca5a42e81a295ae2c4fb81533ef23
Author: Jason Madden <jamadden at gmail.com>
Date: 2020-03-06 (Fri, 06 Mar 2020)
Changed paths:
M appveyor.yml
Log Message:
-----------
Merge pull request #297 from zopefoundation/appveyor-mock
Simplify appveyor.yml.
Commit: 579b5395189beb44033d2a31ec91f33948d1b6f0
https://github.com/zopefoundation/ZODB/commit/579b5395189beb44033d2a31ec91f33948d1b6f0
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M .gitignore
M .travis.yml
M buildout.cfg
M doc/conf.py
M doc/requirements.txt
M tox.ini
Log Message:
-----------
build documentation with tox
Commit: 7b93ab5c8111ad6b09cec4077091b55042f4d514
https://github.com/zopefoundation/ZODB/commit/7b93ab5c8111ad6b09cec4077091b55042f4d514
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M doc/index.rst
Log Message:
-----------
Updated satellite projects links in the documentation. Fixes #236
Commit: 551f6a6c8b3aa598cac3274a3c1cba457a80d6cf
https://github.com/zopefoundation/ZODB/commit/551f6a6c8b3aa598cac3274a3c1cba457a80d6cf
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M doc/conf.py
M doc/index.rst
A doc/introduction.rst
Log Message:
-----------
Splitted documentation index
Commit: be47be34e1d29a9f7171fca2fc996039cbd6bfcc
https://github.com/zopefoundation/ZODB/commit/be47be34e1d29a9f7171fca2fc996039cbd6bfcc
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M README.rst
M doc/README.rst
M doc/index.rst
M setup.py
Log Message:
-----------
Updated documentation links
Commit: ecae906c39f791b9e73ad31f19e40cca23987564
https://github.com/zopefoundation/ZODB/commit/ecae906c39f791b9e73ad31f19e40cca23987564
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M DEVELOPERS.rst
A doc/ConflictResolution.rst
A doc/changelog.rst
A doc/collaborations.rst
A doc/cross-database-references.rst
A doc/developers.rst
A doc/event.rst
A doc/historical_connections.rst
M doc/index.rst
A doc/persistentclass.rst
A doc/subtransactions.rst
A doc/utils.rst
A src/ZODB/ConflictResolution.rst
R src/ZODB/ConflictResolution.txt
R src/ZODB/collaborations.txt
A src/ZODB/cross-database-references.rst
R src/ZODB/cross-database-references.txt
R src/ZODB/event.txt
A src/ZODB/historical_connections.rst
R src/ZODB/historical_connections.txt
A src/ZODB/persistentclass.rst
R src/ZODB/persistentclass.txt
R src/ZODB/subtransactions.txt
M src/ZODB/tests/testUtils.py
M src/ZODB/tests/testconflictresolution.py
M src/ZODB/tests/testcrossdatabasereferences.py
M src/ZODB/tests/testhistoricalconnections.py
M src/ZODB/tests/testpersistentclass.py
A src/ZODB/utils.rst
R src/ZODB/utils.txt
Log Message:
-----------
Brought documentation at the same place
Commit: 5bbd987d55d1d1f4bcd896545f0951d81f3a6665
https://github.com/zopefoundation/ZODB/commit/5bbd987d55d1d1f4bcd896545f0951d81f3a6665
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M doc/index.rst
M doc/requirements.txt
R doc/subtransactions.rst
Log Message:
-----------
Removed subtransactions documentation
Commit: 2657d6b53c9b2d03f323b152be6731e8234f2e9b
https://github.com/zopefoundation/ZODB/commit/2657d6b53c9b2d03f323b152be6731e8234f2e9b
Author: Éloi Rivard <eloi.rivard at aquilenet.fr>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M .gitignore
M .travis.yml
M DEVELOPERS.rst
M README.rst
M buildout.cfg
A doc/ConflictResolution.rst
M doc/README.rst
A doc/changelog.rst
A doc/collaborations.rst
M doc/conf.py
A doc/cross-database-references.rst
A doc/developers.rst
A doc/event.rst
A doc/historical_connections.rst
M doc/index.rst
A doc/introduction.rst
A doc/persistentclass.rst
M doc/requirements.txt
A doc/utils.rst
M setup.py
A src/ZODB/ConflictResolution.rst
R src/ZODB/ConflictResolution.txt
R src/ZODB/collaborations.txt
A src/ZODB/cross-database-references.rst
R src/ZODB/cross-database-references.txt
R src/ZODB/event.txt
A src/ZODB/historical_connections.rst
R src/ZODB/historical_connections.txt
A src/ZODB/persistentclass.rst
R src/ZODB/persistentclass.txt
R src/ZODB/subtransactions.txt
M src/ZODB/tests/testUtils.py
M src/ZODB/tests/testconflictresolution.py
M src/ZODB/tests/testcrossdatabasereferences.py
M src/ZODB/tests/testhistoricalconnections.py
M src/ZODB/tests/testpersistentclass.py
A src/ZODB/utils.rst
R src/ZODB/utils.txt
M tox.ini
Log Message:
-----------
Merge pull request #299 from azmeuk/doc
Some documentation love
Commit: 8eb1927c9719bff9c4908df578e248586c49d4dc
https://github.com/zopefoundation/ZODB/commit/8eb1927c9719bff9c4908df578e248586c49d4dc
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
R log.ini
Log Message:
-----------
Removed log.ini
Commit: e9f793717e80a2b167c2e9304e51be71fe677f86
https://github.com/zopefoundation/ZODB/commit/e9f793717e80a2b167c2e9304e51be71fe677f86
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M .gitignore
M tox.ini
Log Message:
-----------
Use python3 for coverage
Commit: 98808e1622083e6c30239b73a62a7187f55d323a
https://github.com/zopefoundation/ZODB/commit/98808e1622083e6c30239b73a62a7187f55d323a
Author: Éloi Rivard <azmeuk at gmail.com>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M doc/index.rst
Log Message:
-----------
Added newtdb link
Commit: e1f39db4702563a1571297248870648da3a7447b
https://github.com/zopefoundation/ZODB/commit/e1f39db4702563a1571297248870648da3a7447b
Author: Éloi Rivard <eloi.rivard at aquilenet.fr>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
M .gitignore
M tox.ini
Log Message:
-----------
Merge pull request #300 from azmeuk/coverage
Use python3 for coverage
Commit: 96d76caae2e7c93ffb4b383c55de899df5a4585c
https://github.com/zopefoundation/ZODB/commit/96d76caae2e7c93ffb4b383c55de899df5a4585c
Author: Éloi Rivard <eloi.rivard at aquilenet.fr>
Date: 2020-03-17 (Tue, 17 Mar 2020)
Changed paths:
R log.ini
Log Message:
-----------
Merge pull request #301 from azmeuk/cleanups
Removed log.ini
Commit: d8b192550c68a07705ccb9d3fb02a525b90c71b2
https://github.com/zopefoundation/ZODB/commit/d8b192550c68a07705ccb9d3fb02a525b90c71b2
Author: Éloi Rivard <eloi.rivard at aquilenet.fr>
Date: 2020-03-20 (Fri, 20 Mar 2020)
Changed paths:
M doc/index.rst
Log Message:
-----------
Merge pull request #302 from azmeuk/doc-links
Added newtdb link
Commit: ce000346d79694f9e5220fe14927644b31baaf2d
https://github.com/zopefoundation/ZODB/commit/ce000346d79694f9e5220fe14927644b31baaf2d
Author: Stéphane Blondon <stephane.blondon at gmail.com>
Date: 2020-03-26 (Thu, 26 Mar 2020)
Changed paths:
M HISTORY.rst
M README.rst
M doc/articles/ZODB-overview.rst
M doc/articles/multi-zodb-gc.rst
M doc/guide/install-and-run.rst
M doc/guide/transactions-and-threading.rst
M doc/guide/writing-persistent-objects.rst
M doc/index.rst
M doc/reference/storages.rst
M src/ZODB/DemoStorage.py
Log Message:
-----------
Update package links from pypi.python.org to pypi.org
Commit: 4f4102fa52374ac8652e89b80b2b795093f5b9ec
https://github.com/zopefoundation/ZODB/commit/4f4102fa52374ac8652e89b80b2b795093f5b9ec
Author: sblondon <sblondon at users.noreply.github.com>
Date: 2020-03-27 (Fri, 27 Mar 2020)
Changed paths:
M HISTORY.rst
M README.rst
M doc/articles/ZODB-overview.rst
M doc/articles/multi-zodb-gc.rst
M doc/guide/install-and-run.rst
M doc/guide/transactions-and-threading.rst
M doc/guide/writing-persistent-objects.rst
M doc/index.rst
M doc/reference/storages.rst
M src/ZODB/DemoStorage.py
Log Message:
-----------
Merge pull request #304 from sblondon/update_to_pypi.org
Update package links from pypi.python.org to pypi.org
Commit: bb9bf5393294de2d4290bc6c4ea80c35470f747e
https://github.com/zopefoundation/ZODB/commit/bb9bf5393294de2d4290bc6c4ea80c35470f747e
Author: Kirill Smelkov <kirr at nexedi.com>
Date: 2020-03-31 (Tue, 31 Mar 2020)
Changed paths:
M src/ZODB/FileStorage/FileStorage.py
M src/ZODB/tests/testFileStorage.py
Log Message:
-----------
FileStorage: Save committed transaction to disk even if changed data is empty
ZODB tries to avoid saving empty transactions to storage on
`transaction.commit()`. The way it works is: if no objects were changed
during ongoing transaction, ZODB.Connection does not join current
TransactionManager, and transaction.commit() performs two-phase commit
protocol only on joined DataManagers. In other words if no objects were
changed, no tpc_*() methods are called at all on ZODB.Connection at
transaction.commit() time.
This way application servers like Zope/ZServer/ERP5/... can have
something as
try:
# process incoming request
transaction.commit() # processed ok
except:
transaction.abort()
# problem: log + reraise
in top-level code to process requests without creating many on-disk
transactions with empty data changes just because read-only requests
were served.
Everything is working as intended.
However at storage level, FileStorage currently also checks whether
transaction that is being committed also comes with empty data changes,
and _skips_ saving transaction into disk *at all* for such cases, even
if it has been explicitly told to commit the transaction via two-phase
commit protocol calls done at storage level.
This creates the situation, where contrary to promise in
ZODB/interfaces.py(*), after successful tpc_begin/tpc_vote/tpc_finish()
calls made at storage level, transaction is _not_ made permanent,
despite tid of "committed" transaction being returned to caller. In other
words FileStorage, when asked to commit a transaction, even if one with
empty data changes, reports "ok" and gives transaction ID to the caller,
without creating corresponding transaction record on disk.
This behaviour is
a) redundant to application-level avoidance to create empty transaction
on storage described in the beginning, and
b) creates problems:
The first problem is that application that works at storage-level might
be interested in persisting transaction, even with empty changes to
data, just because it wants to save the metadata similarly to e.g.
`git commit --allow-empty`.
The other problem is that an application view and data in database
become inconsistent: an application is told that a transaction was
created with corresponding transaction ID, but if the storage is
actually inspected, e.g. by iteration, the transaction is not there.
This, in particular, can create problems if TID of committed transaction
is reported elsewhere and that second database client does not find the
transaction it was told should exist.
I hit this particular problem with wendelin.core. In wendelin.core,
there is custom virtual memory layer that keeps memory in sync with
data in ZODB. At commit time, the memory is inspected for being dirtied,
and if a page was changed, virtual memory layer joins current
transaction _and_ forces corresponding ZODB.Connection - via which it
will be saving data into ZODB objects - to join the transaction too,
because it would be too late to join ZODB.Connection after 2PC process
has begun(+). One of the format in which data are saved tries to
optimize disk space usage, and it actually might happen, that even if
data in RAM were dirtied, the data itself stayed the same and so nothing
should be saved into ZODB. However ZODB.Connection is already joined
into transaction and it is hard not to join it because joining a
DataManager when the 2PC is already ongoing does not work.
This used to work ok with wendelin.core 1, but with wendelin.core 2 -
where separate virtual filesystem is also connected to the database to
provide base layer for arrays mappings - this creates problem, because
when wcfs (the filesystem) is told to synchronize to view the database
@tid of committed transaction, it can wait forever waiting for that, or
later, transaction to appear on disk in the database, creating
application-level deadlock.
I agree that some more effort might be made at wendelin.core side to
avoid committing transactions with empty data at storage level.
However the most clean way to fix this problem in my view is to fix
FileStorage itself, because if at storage level it was asked to commit
something, it should not silently skip doing so and dropping even non-empty
metadata + returning ok and committed transaction ID to the caller.
As described in the beginning this should not create problems for
application-level ZODB users, while at storage-level the implementation
is now consistently matching interface and common sense.
----
(*) tpc_finish: Finish the transaction, making any transaction changes permanent.
Changes must be made permanent at this point.
...
https://github.com/zopefoundation/ZODB/blob/5.5.1-35-gb5895a5c2/src/ZODB/interfaces.py#L828-L831
(+) https://lab.nexedi.com/kirr/wendelin.core/blob/9ff5ed32/bigfile/file_zodb.py#L788-822
Commit: 47497fc1b22d6196ccca3094885ea1cd177d8b64
https://github.com/zopefoundation/ZODB/commit/47497fc1b22d6196ccca3094885ea1cd177d8b64
Author: Jan-Jaap Driessen <jdriessen at minddistrict.com>
Date: 2020-05-20 (Wed, 20 May 2020)
Changed paths:
M CHANGES.rst
M src/ZODB/FileStorage/FileStorage.py
M src/ZODB/blob.py
M tox.ini
Log Message:
-----------
Fix inconsistent resolution order with zope.interface v5.
Commit: 6dfe525403273139e8ed3fefb559d9b10fe9029a
https://github.com/zopefoundation/ZODB/commit/6dfe525403273139e8ed3fefb559d9b10fe9029a
Author: Jan-Jaap Driessen <jdriessen at minddistrict.com>
Date: 2020-05-20 (Wed, 20 May 2020)
Changed paths:
M .travis.yml
Log Message:
-----------
Add strict iro to travis.
Commit: 14e73eb3d6c273fd3227885de9d83009ba3ebb02
https://github.com/zopefoundation/ZODB/commit/14e73eb3d6c273fd3227885de9d83009ba3ebb02
Author: Jan-Jaap Driessen <jdriessen at minddistrict.com>
Date: 2020-05-20 (Wed, 20 May 2020)
Changed paths:
M src/ZODB/blob.py
Log Message:
-----------
Leave the direct definition of IBlobStorage in place, only expand the definition if necessary inside the __init__.
Commit: 463fc5afaae2005df808e020dcdf6dcd1cb841f2
https://github.com/zopefoundation/ZODB/commit/463fc5afaae2005df808e020dcdf6dcd1cb841f2
Author: Jan-Jaap Driessen <jdriessen at minddistrict.com>
Date: 2020-05-21 (Thu, 21 May 2020)
Changed paths:
M buildout.cfg
M doc/conf.py
M doc/requirements.txt
Log Message:
-----------
The j1m.sphinxautointerface dependency can be dropped, the functionality is now in https://pypi.org/project/sphinxcontrib-zopeext
Commit: 5ce50c3832e994ac2d27e40603f828b365c1caee
https://github.com/zopefoundation/ZODB/commit/5ce50c3832e994ac2d27e40603f828b365c1caee
Author: Julien Muchembled <jm at nexedi.com>
Date: 2020-06-02 (Tue, 02 Jun 2020)
Changed paths:
M .travis.yml
M CHANGES.rst
M buildout.cfg
M doc/conf.py
M doc/requirements.txt
M src/ZODB/FileStorage/FileStorage.py
M src/ZODB/blob.py
M tox.ini
Log Message:
-----------
Merge pull request #310 from zopefoundation/zope-interface-fix-inconsistent-iro
Fix inconsistent resolution order with zope.interface v5.
Commit: 4a6b0283f61427c7a2d8086a271bcdfb1cb53593
https://github.com/zopefoundation/ZODB/commit/4a6b0283f61427c7a2d8086a271bcdfb1cb53593
Author: Julien Muchembled <jm at nexedi.com>
Date: 2020-06-09 (Tue, 09 Jun 2020)
Changed paths:
M src/ZODB/FileStorage/tests.py
M src/ZODB/interfaces.py
M src/ZODB/mvccadapter.py
M src/ZODB/tests/testConnection.py
M src/ZODB/tests/testmvcc.py
Log Message:
-----------
mvccadapter: check if the last TID changed without invalidation
Since commit b5895a5c23309ff2dfe8fd853b838a0c5e349210 ("mvccadapter:
fix race with invalidations when starting a new transaction"),
a ZEO test fails as follows:
File "src/ZEO/tests/drop_cache_rather_than_verify.txt", line 114, in drop_cache_rather_than_verify.txt
Failed example:
conn.root()[1].x
Expected:
6
Got:
1
Earlier in the test, the ZEO server is restarted and then another
client commits. When disconnected, the first client does not receive
invalidations anymore and the connection gets stuck in the past until
there's a new commit after it reconnected. It was possible to make the
test pass with the following patch:
--- a/src/ZEO/ClientStorage.py
+++ b/src/ZEO/ClientStorage.py
@@ -357,6 +357,7 @@ def notify_connected(self, conn, info):
# invalidate our db cache
if self._db is not None:
+ self._db.invalidate(self.lastTransaction(), ())
self._db.invalidateCache()
logger.info("%s %s to storage: %s",
Other implementations like NEO are probably affected the same way.
Rather than changing interfaces in a backward-incompatible way,
this commit revert to the original behaviour, and all the changes
that were done in existing tests are reverted.
However, the interfaces are clarified about the fact that storage
implementations must update at a precise moment the value that is
returned by lastTransaction(): just after invalidate() or
tpc_finish callback.
Commit: 181fa3704b3bc2df0b12672cff24bd4c96f7bbfb
https://github.com/zopefoundation/ZODB/commit/181fa3704b3bc2df0b12672cff24bd4c96f7bbfb
Author: Julien Muchembled <jm at nexedi.com>
Date: 2020-06-10 (Wed, 10 Jun 2020)
Changed paths:
M CHANGES.rst
Log Message:
-----------
changelog: pull request #307
Commit: 3d06c0462d5a511fd81e79caeed310c29c9fc47f
https://github.com/zopefoundation/ZODB/commit/3d06c0462d5a511fd81e79caeed310c29c9fc47f
Author: Julien Muchembled <jm at nexedi.com>
Date: 2020-06-11 (Thu, 11 Jun 2020)
Changed paths:
M CHANGES.rst
M setup.py
Log Message:
-----------
Preparing release 5.6.0
Commit: 8303cf2f7137cefec3799b637440d3871e3ba6a1
https://github.com/zopefoundation/ZODB/commit/8303cf2f7137cefec3799b637440d3871e3ba6a1
Author: Julien Muchembled <jm at nexedi.com>
Date: 2020-06-11 (Thu, 11 Jun 2020)
Changed paths:
M CHANGES.rst
M setup.py
Log Message:
-----------
Back to development: 5.6.1
Commit: e71bb8ce90f23a2a04f5d094288b8e1a4e8db52b
https://github.com/zopefoundation/ZODB/commit/e71bb8ce90f23a2a04f5d094288b8e1a4e8db52b
Author: Jason Madden <jamadden at gmail.com>
Date: 2020-06-12 (Fri, 12 Jun 2020)
Changed paths:
M CHANGES.rst
Log Message:
-----------
Add a change note for #280 [skip ci]
ConnectionPool and ConnectionPool.map both had docstrings and were used by third-party code. People should be warned about this potentially breaking change.
Commit: 22df1fd1141ab0c9db2a163799b87fe3585f15ef
https://github.com/zopefoundation/ZODB/commit/22df1fd1141ab0c9db2a163799b87fe3585f15ef
Author: Jason Madden <jamadden at gmail.com>
Date: 2020-06-12 (Fri, 12 Jun 2020)
Changed paths:
M CHANGES.rst
Log Message:
-----------
Merge pull request #317 from zopefoundation/changenote-280
Add a change note for #280
Commit: 3a493b01087639347283ac78ec41cf6e38f3e9bf
https://github.com/zopefoundation/ZODB/commit/3a493b01087639347283ac78ec41cf6e38f3e9bf
Author: Kirill Smelkov <kirr at nexedi.com>
Date: 2020-07-31 (Fri, 31 Jul 2020)
Changed paths:
M src/ZODB/Connection.py
M src/ZODB/POSException.py
M src/ZODB/interfaces.py
M src/ZODB/mvccadapter.py
M src/ZODB/tests/testZODB.py
M src/ZODB/tests/testmvcc.py
M src/ZODB/transact.py
Log Message:
-----------
Kill leftovers of pre-MVCC read conflicts
In the early days, before MVCC was introduced, ZODB used to raise
ReadConflictError on access to object that was simultaneously changed by
another client in concurrent transaction. However, as
doc/articles/ZODB-overview.rst says
Since Zope 2.8 ZODB has implemented **Multi Version Concurrency Control**.
This means no more ReadConflictErrors, each transaction is guaranteed to be
able to load any object as it was when the transaction begun.
So today the only way to get a ReadConflictError should be
1) at commit time for an object that was requested to stay unchanged
via checkCurrentSerialInTransaction, and
2) at plain access time, if a pack running simultaneously to current
transaction, removes object revision that we try to load.
The second point is a bit unfortunate, since when load discovers that
object was deleted or not yet created, it is logically more clean to
raise POSKeyError. However due to backward compatibility we still want
to raise ReadConflictError in this case - please see comments added to
MVCCAdapter for details.
Anyway, let's remove leftovers of handling regular read-conflicts from
pre-MVCC era:
Adjust docstring of ReadConflictError to explicitly describe that this
error can only happen at commit time for objects requested to be
current, or at plain access if pack is running simultaneously under
connection foot.
There were also leftover code, comment and test bits in Connection,
interfaces, testmvcc and testZODB, that are corrected/removed
correspondingly. testZODB actually had ReadConflictTests that was
completely deactivated: commit b0f992fd ("Removed the mvcc option..."; 2007)
moved read-conflict-on-access related tests out of ZODBTests, but did not
activated moved parts at all, because as that commit says when MVCC is
always on unconditionally, there is no on-access conflicts:
Removed the mvcc option. Everybody wants mvcc and removing us lets us
simplify the code a little. (We'll be able to simplify more when we
stop supporting versions.)
Today, if I try to manually activate that ReadConflictTests via
@@ -637,6 +637,7 @@ def __init__(self, poisonedjar):
def test_suite():
return unittest.TestSuite((
unittest.makeSuite(ZODBTests, 'check'),
+ unittest.makeSuite(ReadConflictTests, 'check'),
))
if __name__ == "__main__":
it fails in dumb way showing that this tests were unmaintained for ages:
Error in test checkReadConflict (ZODB.tests.testZODB.ReadConflictTests)
Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 320, in run
self.setUp()
File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/testZODB.py", line 451, in setUp
ZODB.tests.utils.TestCase.setUp(self)
AttributeError: 'module' object has no attribute 'utils'
Since today ZODB always uses MVCC and there is no way to get
ReadConflictError on concurrent plain read/write access, those tests
should be also gone together with old pre-MVCC way of handling
concurrency.
/cc @jimfulton
/reviewed-on https://github.com/zopefoundation/ZODB/pull/320
/reviewed-by @jamadden
Commit: 4aa6218584edc7412ed24470977c7b86243c61cf
https://github.com/zopefoundation/ZODB/commit/4aa6218584edc7412ed24470977c7b86243c61cf
Author: Julien Muchembled <jm at nexedi.com>
Date: 2020-08-19 (Wed, 19 Aug 2020)
Changed paths:
M src/ZODB/tests/BasicStorage.py
Log Message:
-----------
Relax assertion in check_tid_ordering_w_commit test
It is pointless for lastTransaction() to block until it is allowed to
return the TID of a transaction that has just been committed, because
it may still not be the real last TID (e.g. for some storage
implementations, invalidations are received from a shared server
via the network). While invalidations are still being processed,
it's fine to return immediately with the previous last TID.
This was clarified in commit 4a6b0283f61427c7a2d8086a271bcdfb1cb53593
("mvccadapter: check if the last TID changed without invalidation").
See pull request #316
Commit: d08185b3b0a37446fbfa76fea2e2ec1b4226fd70
https://github.com/zopefoundation/ZODB/commit/d08185b3b0a37446fbfa76fea2e2ec1b4226fd70
Author: Jérome Perrin <jerome at nexedi.com>
Date: 2020-08-26 (Wed, 26 Aug 2020)
Changed paths:
M buildout.cfg
Log Message:
-----------
buildout: pin sphinxcontrib-websupport to < 1.2 on python2
Starting from 1.2.0 sphinxcontrib-websupport officially only supports python3.
Since 1.2.4 it depends on sphinxcontrib-serializinghtml which can not even be
imported on python2
Commit: 067c9183532c60f8047c3bc8efa688aaa48f4198
https://github.com/zopefoundation/ZODB/commit/067c9183532c60f8047c3bc8efa688aaa48f4198
Author: Jérome Perrin <jerome at nexedi.com>
Date: 2020-08-26 (Wed, 26 Aug 2020)
Changed paths:
M buildout.cfg
Log Message:
-----------
buildout: define versions constraints with [versions]
For consistency, defined all versions constraints in version section.
Also don't mention python34 since ZODB no longer supports python3.4
Commit: b4f233cd83f0b9a6138989b3b1f8f92eeadcfefe
https://github.com/zopefoundation/ZODB/commit/b4f233cd83f0b9a6138989b3b1f8f92eeadcfefe
Author: Jérome Perrin <jerome at nexedi.com>
Date: 2020-08-31 (Mon, 31 Aug 2020)
Changed paths:
M buildout.cfg
Log Message:
-----------
Merge pull request #327 from perrinjerome/doc-sphinx-python3
Fix requirements for sphinx on python2
Commit: c1e080520a466d88c2835b7489d01b2dc7e31c2e
https://github.com/zopefoundation/ZODB/commit/c1e080520a466d88c2835b7489d01b2dc7e31c2e
Author: Kirill Smelkov <kirr at nexedi.com>
Date: 2020-08-31 (Mon, 31 Aug 2020)
Changed paths:
M src/ZODB/interfaces.py
Log Message:
-----------
interface: Require invalidations to be called with full set of objects and not to skip transactions
Currently invalidate documentation is not clear whether it should be
called for every transaction and whether it should include full set of
objects created/modified by that transaction. Until now this was working
relatively well for the sole purpose of invalidating client ZEO cache,
because for that particular task it is relatively OK not to include just
created objects into invalidation messages, and even to completely skip
sending invalidation if transaction only create - not modify - objects.
Due to this fact the workings of the client cache was indifferent to the
ambiguity of the interface.
In 2016 skipping transactions with only created objects was reconsidered
as bug and fixed in ZEO5 because ZODB5 relies more heavily on MVCC
semantic and needs to be notified about every transaction committed to
storage to be able to properly update ZODB.Connection view:
https://github.com/zopefoundation/ZEO/commit/02943acd#diff-52fb76aaf08a1643cdb8fdaf69e37802L889-R834
https://github.com/zopefoundation/ZEO/commit/9613f09b
However just-created objects were not included into invalidation
messages until, hopefully, recently:
https://github.com/zopefoundation/ZEO/pull/160
As ZODB is started to be used more widely in areas where it was not
traditionally used before, the ambiguity in invalidate interface and the
lack of guarantees - for any storage - to be notified with full set of
information, creates at least the following problems:
- a ZODB client (not necessarily native ZODB/py client) can maintain
raw cache for the storage. If such client tries to load an oid at
database view when that object did not existed yet, gets "no object"
reply and stores that information into raw cache, to properly invalidate
the cache it needs an invalidation message from ZODB server that
*includes* created object.
- tools like `zodb watch` [1,2,3] don't work properly (give incorrect output)
if not all objects modified/created by a transaction are included into
invalidation messages.
- similarly to `zodb watch`, a monitoring tool, that would want to be
notified of all created/modified objects, won't see full
database-change picture, and so won't work properly without knowing
which objects were created.
- wendelin.core 2 - which builds data from ZODB BTrees and data objects
into virtual filesystem - needs to get invalidation messages with both
modified and created objects to properly implement its own lazy
invalidation and isolation protocol for file blocks in OS cache: when
a block of file is accessed, all clients, that have this block mmaped,
need to be notified and asked to remmap that block into particular
revision of the file depending on a client's view of the filesystem and
database [4,5].
To compute to where a client needs to remmap the block, WCFS server
(that in turn acts as ZODB client wrt ZEO/NEO server), needs to be able
to see whether client's view of the filesystem is before object creation
(and then ask that client to pin that block to hole), or after creation
(and then ask the client to pin that block to corresponding revision).
This computation needs ZODB server to send invalidation messages in
full: with both modified and just created objects.
Also:
- the property that all objects - both modified and just created -
are included into invalidation messages is required and can help to
remove `next_serial` from `loadBefore` return in the future.
This, in turn, can help to do 2x less SQL queries in loadBefore for
NEO and RelStorage (and maybe other storages too):
https://github.com/zopefoundation/ZODB/issues/318#issuecomment-657685745
Current state of storages with respect to new requirements:
- ZEO: does not skip transactions, but includes only modified - not
created - objects. This is fixed by https://github.com/zopefoundation/ZEO/pull/160
- NEO: already implements the requirements in full
- RelStorage: already implements the requirements in full, if I
understand correctly:
https://github.com/zodb/relstorage/blob/3.1.2-1-gaf57d6c/src/relstorage/adapters/poller.py#L28-L145
While editing invalidate documentation, use the occasion to document
recently added property that invalidate(tid) is always called before
storage starts to report its lastTransaction() ≥ tid - see 4a6b0283
(mvccadapter: check if the last TID changed without invalidation).
/cc @jimfulton, @jamadden, @jmuchemb, @vpelletier, @arnaud-fontaine, @gidzit, @klawlf82, @jwolf083
/reviewed-on https://github.com/zopefoundation/ZODB/pull/319
/reviewed-by @dataflake
/reviewed-by @jmuchemb
[1] https://lab.nexedi.com/kirr/neo/blob/049cb9a0/go/zodb/zodbtools/watch.go
[2] https://lab.nexedi.com/kirr/neo/commit/e0d59f5d
[3] https://lab.nexedi.com/kirr/neo/commit/c41c2907
[4] https://lab.nexedi.com/kirr/wendelin.core/blob/1efb5876/wcfs/wcfs.go#L94-182
[5] https://lab.nexedi.com/kirr/wendelin.core/blob/1efb5876/wcfs/client/wcfs.h#L20-71
Commit: 1da7a93700bfa53c3558c9033a5ab0701d544f7c
https://github.com/zopefoundation/ZODB/commit/1da7a93700bfa53c3558c9033a5ab0701d544f7c
Author: Jérome Perrin <jerome at nexedi.com>
Date: 2020-09-04 (Fri, 04 Sep 2020)
Changed paths:
M .travis.yml
Log Message:
-----------
.travis.yml: pin setuptools to 49.6 on pypy3
setutools 50.0.0 does not seem compatible with the pypy3 used on travis.
Commit: 25f66d34f710679bef37a14f2acf775b5002a726
https://github.com/zopefoundation/ZODB/commit/25f66d34f710679bef37a14f2acf775b5002a726
Author: Jérome Perrin <jerome at nexedi.com>
Date: 2020-09-04 (Fri, 04 Sep 2020)
Changed paths:
M doc/tutorial.rst
Log Message:
-----------
doc: fix typo _p_changed
To manually mark an object modified, `_p_changed` attribute must be set, no `_p_changed__`
Commit: 79209ea99180b5182e6b22fc978b07f378eb4f5d
https://github.com/zopefoundation/ZODB/commit/79209ea99180b5182e6b22fc978b07f378eb4f5d
Author: ale-rt <alessandro.pisa at gmail.com>
Date: 2020-09-22 (Tue, 22 Sep 2020)
Changed paths:
M .gitignore
M .travis.yml
M CHANGES.rst
M DEVELOPERS.rst
M HISTORY.rst
M README.rst
M appveyor.yml
M buildout.cfg
A doc/ConflictResolution.rst
M doc/README.rst
M doc/articles/ZODB-overview.rst
M doc/articles/multi-zodb-gc.rst
A doc/changelog.rst
A doc/collaborations.rst
M doc/conf.py
A doc/cross-database-references.rst
A doc/developers.rst
A doc/event.rst
M doc/guide/install-and-run.rst
M doc/guide/transactions-and-threading.rst
M doc/guide/writing-persistent-objects.rst
A doc/historical_connections.rst
M doc/index.rst
A doc/introduction.rst
A doc/persistentclass.rst
M doc/reference/storages.rst
M doc/requirements.txt
M doc/tutorial.rst
A doc/utils.rst
R log.ini
M setup.py
A src/ZODB/ConflictResolution.rst
R src/ZODB/ConflictResolution.txt
M src/ZODB/Connection.py
M src/ZODB/DemoStorage.py
M src/ZODB/FileStorage/FileStorage.py
M src/ZODB/FileStorage/tests.py
M src/ZODB/POSException.py
M src/ZODB/blob.py
R src/ZODB/collaborations.txt
A src/ZODB/cross-database-references.rst
R src/ZODB/cross-database-references.txt
R src/ZODB/event.txt
A src/ZODB/historical_connections.rst
R src/ZODB/historical_connections.txt
M src/ZODB/interfaces.py
M src/ZODB/mvccadapter.py
A src/ZODB/persistentclass.rst
R src/ZODB/persistentclass.txt
R src/ZODB/subtransactions.txt
M src/ZODB/tests/BasicStorage.py
M src/ZODB/tests/testCache.py
M src/ZODB/tests/testConnection.py
M src/ZODB/tests/testConnectionSavepoint.py
M src/ZODB/tests/testFileStorage.py
M src/ZODB/tests/testThreadedShutdown.py
M src/ZODB/tests/testUtils.py
M src/ZODB/tests/testZODB.py
M src/ZODB/tests/testconflictresolution.py
M src/ZODB/tests/testcrossdatabasereferences.py
M src/ZODB/tests/testhistoricalconnections.py
M src/ZODB/tests/testmvcc.py
M src/ZODB/tests/testpersistentclass.py
M src/ZODB/transact.py
A src/ZODB/utils.rst
R src/ZODB/utils.txt
M tox.ini
Log Message:
-----------
Merge remote-tracking branch 'origin/master' into fix_UnboundLocalError
Compare: https://github.com/zopefoundation/ZODB/compare/32ebc3f6bccf...79209ea99180
More information about the checkins
mailing list