[Checkins] SVN: zdgbook/trunk/ stx to rst
Baiju M
baiju.m.mail at gmail.com
Tue Feb 17 11:26:51 EST 2009
Log message for revision 96643:
stx to rst
Changed:
D zdgbook/trunk/ZODBPersistentComponents.stx
A zdgbook/trunk/source/ZODBPersistentComponents.rst
U zdgbook/trunk/source/index.rst
-=-
Deleted: zdgbook/trunk/ZODBPersistentComponents.stx
===================================================================
--- zdgbook/trunk/ZODBPersistentComponents.stx 2009-02-17 13:16:30 UTC (rev 96642)
+++ zdgbook/trunk/ZODBPersistentComponents.stx 2009-02-17 16:26:51 UTC (rev 96643)
@@ -1,842 +0,0 @@
-Chapter 4: ZODB Persistent Components
-
- Most Zope components live in the Zope Object DataBase (ZODB).
- Components that are stored in ZODB are said to be *persistent*.
- Creating persistent components is, for the most part, a trivial
- exercise, but ZODB does impose a few rules that persistent
- components must obey in order to work properly. This chapter
- describes the persistence model and the interfaces that persistent
- objects can use to live inside the ZODB.
-
- % Anonymous User - July 9, 2002 8:38 am:
- In a SAMS book, "ZOPE Web Application Construction Kit" by Martina Brockmann etc, page 41, it says "... ZODB
- is not suitable for storing large amounts of data."
- Is it true? If it is, we are in trouble because Zope stores everything in it. This means that Zope is not
- suitable for large site. Can somebody please clarify this? Thanks.
-
- % Anonymous User - July 9, 2002 8:58 am:
- Sure. This statement is actually completely false. ZODB is very capable of storing large amounts of data. But
- like any sort of database you need to be careful to put the right kinds of data structures in the right
- places for maximum efficiency. Also, you may want to consider a different storage than FileStorage if you
- have lots of data because FileStorage keeps all data in a single file, which can be problematic on systems
- which do not support large files. Additionally, ZODB recovery tools are currently not as robust as, say,
- Oracle recovery tools, so if your database does get corrupted due to hardware failure it may not be a
- recoverable error and you may need to restore from backup.
- That said, there is nothing stopping you from using Oracle or the filesystem to store all of your data just
- like other application servers. Zope lets you do this. You don't need to use ZODB. The equation you use above
- is inaccurate.
-
- % Anonymous User - July 9, 2002 9:17 am:
- Thanks for your quick reply.
- How can I find more info about:
- (1) any guideline regarding "to put the right kinds of data structures in the right places for maximum
- efficiency;"
- (2) how Zope works with other application servers if I store static html pages and graphics files on those
- servers.
- Thanks again.
-
- % Anonymous User - July 9, 2002 9:57 am:
- Ask a detailed question including what kinds of data you want to store and which other servers you want to
- integrate with on the Zope maillist (zope at zope.org). See the Resources link of Zope.org for more maillist
- info.
-
- % Anonymous User - Jan. 16, 2003 1:40 am:
- More importantly, how do you configure zope to use an Oracle database instead of ZODB? I can find information
- on DCOracleStorage stuffs, but nothing about configuring Zope to startup and point to an Oracle database.
-
- % axxackall - Apr. 5, 2003 11:09 am:
- I know several projects where developers had to re-write half of the system just because it was based on
- object persistence and POET (or other) ODBMS. It is a fact that enterprise users do not like ODBMS with
- consequently closed architecture. They want RDBMS with ability to integrate the web portal with other
- corporate infrmation sources. How to do that with ZODB? No way. I've tried to find any information about any
- attempts to get rid from ZODB on the backend of Zope and found that virtually nobody is working currently on
- it. If anyone is interesting in it, particularly in substituting ZODB by PostgreSQL, please send me email:
- axxackall at yahoo.com as such project is too much for one busy developer.
-
- % Anonymous User - Dec. 22, 2003 4:17 pm:
- "get rid from ZODB" is not going to happen. Zope is a ZODB application.
- What you should probably look at is APE, which (properly configured) will transparently
- store zodb data in the RDBMS of your choice. It is not yet stable but some people are already using it.
-
- % Anonymous User - June 27, 2005 7:53 pm:
- test
-
- Persistent Objects
-
- Persistent objects are Python objects that live for a long
- time. Most objects are created when a program is run and die when
- the program finishes. Persistent objects are not destroyed when
- the program ends, they are saved in a database.
-
- % Anonymous User - June 27, 2005 7:52 pm:
- test
-
- A great benefit of persistent objects is their transparency. As a
- developer, you do not need to think about loading and unloading
- the state of the object from memory. Zope's persistent machinery
- handles all of that for you.
-
- % Anonymous User - Feb. 21, 2002 7:47 pm - A note on __module_aliases__. It almost never works. Everytime I've wanted to do this, it has no effect. References to classes are to pervasive given the standard Zopish ways of doing things.
-
- % mcdonc - Mar. 2, 2002 4:58 pm - Sorry to hear you're having difficulties with it, but it does work. I've used it many times.
-
- This is also a great benefit for application designers; you do not
- need to create your own kind of "data format" that gets saved to a
- file and reloaded again when your program stops and starts.
- Zope's persistence machinery works with *any* kind of Python
- objects (within the bounds of a few simple rules) and as your
- types of objects grow, your database simply grows transparently
- with it.
-
- % Anonymous User - Dec. 9, 2001 3:47 pm - The topic came up on the maillist of how a persistent object "finds" its class. A persistent object has a reference to its class' fully-qualified name including at least one module name. For example, an instance of the DTMLMethod class finds its class at 'Products.OFSP.DTMLMethod.DTMLMethod', where 'Products' and 'OFSP' are Python packages, the first 'DTMLMethod' refers to a module file, and the last 'DTMLMethod' refers to the classname. If you change the location of a class within the filesystem (for example, if you moved the above Python class definition to 'Foo.Bar.DTMLMethod', instances of the original class that were stored in the ZODB will not be loadable. There is a facility named __module_aliases__ in Zope for aliasing a module to more than one name, although there is no such facility for use of ZODB outside Zope. *Need more docs on __module_aliases__* - chrism.
-
- % Anonymous User - Dec. 9, 2001 3:53 pm - Example of __module_aliases__ follows. It works by putting a tuple of tuples in your Product's __init__.py module in a __module_aliases__ attribute at module scope. For example, PythonScripts have the following __module_aliases__ attribute. from Shared.DC import Scripts __module_aliases__ = ( ('Products.PythonScripts.Script', Scripts.Script), ('Products.PythonScripts.Bindings', Scripts.Bindings), ('Products.PythonScripts.BindingsUI', Scripts.BindingsUI),) .. this maps the module that *used* to be at Products.PythonScripts.Script to the module that is *now* at Scripts.Script, etc. This only works with modules and not with classes or other types.
-
- % Anonymous User - Jan. 17, 2002 10:12 am - change/your types of objects grow/your number of objects grow/
-
- Persistence Example
-
- Here is a simple example of using ZODB outside of Zope. If all
- you plan on doing is using persistent objects with Zope, you can
- skip this section if you wish.
-
- % Anonymous User - Oct. 21, 2002 10:16 am:
- well, it is really funny that the only example about persistence is not valid inside Zope. After reading this
- chapter, I still cannot use persistence. I think this chapter should explain how to use persistance inside
- Zope
-
- % Anonymous User - Nov. 8, 2002 10:27 pm:
- Maybe the ZDG should be split into a) Product developement inside ZODB and
- b) Advanced Developers Guide?
-
- The first thing you need to do to start working with ZODB is to
- create a "root object". This process involves first opening a
- "storage" , which is the actual backend storage location for your
- data.
-
- ZODB supports many pluggable storage back-ends, but for the
- purposes of this article we're going to show you how to use the
- 'FileStorage' back-end storage, which stores your object data in a
- file. Other storages include storing objects in relational
- databases, Berkeley databases, and a client to server storage that
- stores objects on a remote storage server.
-
- % Anonymous User - Nov. 8, 2002 10:29 pm:
- Berkely db: URL?
-
- % Anonymous User - Nov. 16, 2002 3:40 am:
- A little more information or pointers to info on the different pluggable storage back-ends would go down a
- treat
-
- To set up a ZODB, you must first install it. ZODB comes with
- Zope, so the easiest way to install ZODB is to install Zope and
- use the ZODB that comes with your Zope installation. For those of
- you who don't want all of Zope, but just ZODB, see the
- instructions for downloading ZODB from the "ZODB web
- page":http://www.zope.org/Wikis/ZODB/FrontPage.
-
- After installing ZODB, you can start to experiment with it right
- from the Python command line interpreter. If you've installed
- Zope, before running this set of commands, shut down your Zope server,
- and "cd" to the "lib/python" directory of your Zope instance.
- If you're using a "standalone" version of ZODB, you likely don't need
- to do this, and you'll be able to use ZODB by importing it from a
- standard Python package directory. In either case, try the
- following set of commands::
-
- chrism at saints:/opt/zope/lib/python$ python
- Python 2.1.1 (#1, Aug 8 2001, 21:17:50)
- [GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2
- Type "copyright", "credits" or "license" for more information.
- >>> from ZODB import FileStorage, DB
- >>> storage = FileStorage.FileStorage('mydatabase.fs')
- >>> db = DB( storage )
- >>> connection = db.open()
- >>> root = connection.root()
-
- Here, you create storage and use the 'mydatabse.fs' file to store
- the object information. Then, you create a database that uses
- that storage.
-
- % Anonymous User - July 18, 2002 1:36 pm:
- s/mydatabse\.fs/mydatabase\.fs/
-
- Next, the database needs to be "opened" by calling the 'open()'
- method. This will return a connection object to the database.
- The connection object then gives you access to the 'root' of the
- database with the 'root()' method.
-
- The 'root' object is the dictionary that holds all of your
- persistent objects. For example, you can store a simple list of
- strings in the root object::
-
- root['employees'] = ['Bob', 'Mary', 'Jo']
-
- Now, you have changed the persistent database by adding a new
- object, but this change is so far only temporary. In order to
- make the change permanent, you must commit the current
- transaction::
-
- get_transaction().commit()
-
- Transactions are ways to make a lot of changes in one atomic
- operation. In a later article, we'll show you how this is a very
- powerful feature. For now, you can think of committing
- transactions as "checkpoints" where you save the changes you've
- made to your objects so far. Later on, we'll show you how to
- abort those changes, and how to undo them after they are
- committed.
-
- If you had used a relational database, you would have had to issue
- a SQL query to save even a simple python list like the above
- example. You would have also needed some code to convert a SQL
- query back into the list when you wanted to use it again. You
- don't have to do any of this work when using ZODB. Using ZODB is
- almost completely transparent, in fact, ZODB based programs often
- look suspiciously simple!
-
- Working with simple python types is useful, but the real power of
- ZODB comes out when you store your own kinds of objects in the
- database. For example, consider a class that represents a
- employee::
-
- from Persistence import Persistent
-
- class Employee(Persistent):
-
- def setName(self, name):
- self.name = name
-
- % Anonymous User - Oct. 16, 2002 7:43 am:
- I get the message
- Error Value: exceptions.ImportError on import of "Persistent" from "Persistence" is unauthorized in '', at
- line 15, column 11
- I am the zope administrator and I am running it in a Win98, therefore shouldn't be problems with the
- authorization.
- I am running it in a Python script that I call from a dtml page. Is there any problem with this? somebody can
- tell me why I get this problem?
-
- Calling 'setName' will set a name for the employee. Now, you can
- put Employee objects in your database::
-
- for name in ['Bob', 'Mary', 'Joe']:
- employee = Employee()
- employee.setName(name)
- root['employees'].append(employee)
-
- get_transaction().commit()
-
- Don't forget to call 'commit()', so that the changes you have made
- so far are committed to the database, and a new transaction is
- begun.
-
- % Anonymous User - Apr. 10, 2003 5:03 pm:
- This example seems to break the fourth rule as mentioned below. You are adding to the root's employees entry,
- a list, which is a mutable thing.
- At least, this is true when running the Python interpreter with a standalone ZODB.
-
- % Anonymous User - Nov. 22, 2003 11:44 pm:
- No, it is saying that you must notify the database to persist each change after it has been made, because the
- default Python list type does not know about persistence. If you use the types provided with ZODB, then
- changes will be persisted automatically. The type is smart enough to update the database when the in-memory
- copy changes.
-
- % Anonymous User - Mar. 15, 2004 9:33 am:
- well said
-
- % Anonymous User - Oct. 12, 2004 5:31 pm:
- well...
-
- Persistent Rules
-
- There are a few rules that must be followed when your objects are
- persistent.
-
- o Your objects, and their attributes, must be "pickleable".
-
- o Your object cannot have any attributes that begin with '_p_'.
-
- o Attributes of your object that begin with '_v_' are "volatile"
- and are not saved to the database (see next section).
-
- o You must explicitly signal any changes made to mutable
- attributes (such as instances, lists, and dictionaries) or use
- persistent versions of mutable objects, like
- 'ZODB.PersistentMapping' (see below for more information on
- 'PersistentMapping'.)
-
- In this section, we'll look at each of these special rules one by
- one.
-
- The first rules says that your objects must be pickleable. This
- means that they can be serialized into a data format with the
- "pickle" module. Most python data types (numbers, lists,
- dictionaries) can be pickled. Code objects (method, functions,
- classes) and file objects (files, sockets) *cannot* be pickled.
- Instances can be persistent objects if:
-
- o They subclass 'Persistence.Persistent'
-
- o All of their attributes are pickleable
-
- % Anonymous User - Nov. 8, 2002 10:57 pm:
- The result of a ZSQL Method can not be pickled (thanks to a class r():pass)
-
- % Anonymous User - Apr. 10, 2003 5:03 pm:
- s/rules/rule/
-
- The second rule is that none of your objects attributes can begin
- with '_p_'. For example, '_p_b_and_j' would be an illegal object
- attribute. This is because the persistence machinery reserves all
- of these names for its own purposes.
-
- The third rule is that all object attributes that begin with
- '_v_' are "volatile" and are not saved to the database. This
- means that as long as the persistent object is in Zope memory
- cache, volatile attributes can be used. When the object is
- deactivated (removed from memory) volatile attributes are thrown
- away.
-
- Volatile attributes are useful for data that is good to cache for
- a while but can often be thrown away and easily recreated. File
- connections, cached calculations, rendered templates, all of these
- kinds of things are useful applications of volatile
- attributes. You must exercise care when using volatile attributes.
- Since you have little control over when your objects are moved in
- and out of memory, you never know when your volatile attributes
- may disappear.
-
- % poster - May 13, 2002 9:43 am:
- I assume that you can count on a volatile attribute remaining within the life of a method call that creates
- it. What about within a transaction? In general, while I understand there will be a point at which you can no
- longer rely on the existence of a volatile attribute, when *can* you rely on it?
-
- % reiman - Aug. 16, 2002 12:13 pm:
- I also just learned that _v_ attributes are thread-specific. This too should be mentioned (and explained)
- here.
-
- % beyond - Oct. 5, 2002 10:44 am:
- Within one transaction you can rely on _v_ attributes (afaik).
- Each thread gets its own transaction. So another thread -> another _v_ attribute
- -> you can't rely on it in different transactions. Tansactions are kept in a pool
- and objects are cached so sometimes when accessing an object with a new
- request (which gets the old transaction out of the pool) the object is still cached
- and it won't get loaded out of ZODB and then __setstate__ won't get called and
- finally the _v_ attribute remains. But this is of course not relyable. I'm just
- telling you this special case because recently I got some strange errors which were
- caused by this behavior.
-
- The fourth rule is that you must signal changes to mutable types.
- This is because persistent objects can't detect when mutable types
- change, and therefore, doesn't know whether or not to save the
- persistent object or not.
-
- For example, say you had a list of names as an attribute of your object
- called 'departments' that you changed in a method called
- 'addDepartment'::
-
- class DepartmentManager(Persistent):
-
- def __init__(self):
- self.departments = []
-
- def addDepartment(self, department):
- self.departments.append(department)
-
- When you call the 'addDepartment' method you change a mutable
- type, 'departments' but your persistent object will not save that
- change.
-
- There are two solutions to this problem. First, you can assign a
- special flag, '_p_changed'::
-
- def addDepartment(self, department):
- self.department.append(department)
- self._p_changed = 1
-
- % Anonymous User - Mar. 24, 2003 4:23 pm:
- It says there are "two solutions", and then, "First...". What is the second solution?
-
- % Anonymous User - Nov. 8, 2002 11:05 pm:
- well, these are simple types. i had a (recursive) attribute
- self.tree = [..., {'sub': <a subtree> ...} ...]
- and sometimes (unreproducable) it didnt update in lieu of self._p_changed=1.
- Explain.
-
- Remember, '_p_' attributes do something special to the persistence
- machinery and are reserved names. Assigning 1 to '_p_changed'
- tells the persistence machinery that you changed the object, and
- that it should be saved.
-
- Another technique is to use the mutable attribute as though it
- were immutable. In other words, after you make changes to a
- mutable object, reassign it::
-
- def addDepartment(self, department):
- departments = self.departments
- departments.append(department)
- self.department = departments
-
- % Anonymous User - Aug. 3, 2002 3:59 pm:
- Reassigning self.departments stores the entire list again. ZODB can know nothing about your intent. If you
- have an often-changing list like this, it'd likely be better to store it as a BTree (see
- lib/python/BTrees/Interfaces.py in the source).
-
- % Anonymous User - Aug. 3, 2002 3:45 pm:
- which technique is more efficient? When you tag a mutable object (i.e. list) with _p_changed = 1; what does
- ZODB do to change the list? Does it commit the entire list all over again or does it just commit the new list
- element? Using a mutable as though it were immutable screams inefficiency to me... what if the list is HUGE
- (i.e. 1024 instances of a persistant class)? For the assignment: departments = self.departments, does ZODB
- pull the entire list out of storage or does it just do a shallow copy? I assume shallow... please advise.
-
- % bernddorn - Dec. 1, 2001 7:46 am - there are some mistakes in the example code: "self.department = departments" should be "self.departments = departments" further above, the same mistake appears.
-
- Here, the 'self.departments' attribute was re-assigned at the end
- of the function to the "working copy" object 'departments'. This
- technique is cleaner because it doesn't have any explicit
- '_p_changed' settings in it, but this implicit triggering of the
- persistence machinery should always be understood, otherwise use
- the explicit syntax.
-
- A final option is to use persistence-aware mutable attributes such
- as 'PersistentMapping', and 'IOBTree'. 'PersistentMapping' is a
- mapping class that notifies ZODB when you change the mapping. You
- can use instances of 'PersistentMapping' in place of standard
- Python dictionaries and not worry about signaling change by
- reassigning the attribute or using '_p_changed'. Zope's Btree
- classes are also persistent-aware mutable containers. This
- solution can be cleaner than using mutable objects immutably, or
- signaling change manually assuming that there is a
- persistence-aware class available that meets your needs.
-
- Transactions and Persistent Objects
-
- When changes are saved to ZODB, they are saved in a *transaction*.
- This means that either all changes are saved, or none are saved.
- The reason for this is data consistency. Imagine the following
- scenario:
-
- 1. A user makes a credit card purchase at the sandwich.com website.
-
- 2. The bank debits their account.
-
- 3. An electronic payment is made to sandwich.com.
-
- Now imagine that an error happens during the last step of this
- process, sending the payment to sandwich.com. Without
- transactions, this means that the account was debited, but the
- payment never went to sandwich.com! Obviously this is a bad
- situation. A better solution is to make all changes in a
- transaction:
-
- 1. A user makes a credit card purchase at the sandwich.com website.
-
- 2. The transaction begins
-
- 3. The bank debits their account.
-
- 4. An electronic payment is made to sandwich.com.
-
- 5. The transaction commits
-
- Now, if an error is raised anywhere between steps 2 and 5, *all*
- changes made are thrown away, so if the payment fails to go to
- sandwich.com, the account won't be debited, and if debiting the
- account raises an error, the payment won't be made to
- sandwich.com, so your data is always consistent.
-
- When using your persistent objects with Zope, Zope will automatically
- *begin* a transaction when a web request is made, and *commit* the
- transaction when the request is finished. If an error occurs at any
- time during that request, then the transaction is *aborted*, meaning
- all the changes made are thrown away.
-
- If you want to *intentionally* abort a transaction in the middle
- of a request, then just raise an error at any time. For example,
- this snippet of Python will raise an error and cause the transaction
- to abort::
-
- raise SandwichError('Not enough peanut butter.')
-
- A more likely scenario is that your code will raise an exception
- when a problem arises. The great thing about transactions is that
- you don't have to include cleanup code to catch exceptions and
- undo everything you've done up to that point. Since the
- transaction is aborted the changes made in the transaction will
- not be saved.
-
- Because Zope does transaction management for you, most of the time you
- do not need to explicitly begin, commit or abort your own
- transactions. For more information on doing transaction management
- manually, see the links at the end of this chapter that lead to more
- detailed tutorials of doing your own ZODB programming.
-
- % Anonymous User - Jan. 4, 2002 9:14 am - The text should probably mention that you have to let the exception propagate "right out of Zope" for the "rollback" to occur in Zope (of course). Otherwise, it seems to be the case that if the exception is to be handled within a Zope Product (so that a user of the application doesn't see the standard error page), then an explicit transaction abort should be performed in the exception handler in question.
-
- % peterb - Aug. 16, 2002 3:48 am:
- It should be even more specific and mention that if the exception is caught, whether in a product, in DTML or
- in a script there is no automatic rollback.
- You need to call get_transaction.abort() in your exception handler, unless you rethrow the exception and you
- know it won't get caught.
- This is actually guesswork due to lacking docs, I just happen to have the problem right now.
-
- Subtransactions
-
- Zope waits until the transaction is committed to save all the
- changes to your objects. This means that the changes are saved in
- memory. If you try to change more objects than you have memory in
- your computer, your computer will begin to swap and thrash, and
- maybe even run you out of memory completely. This is bad. The
- easiest solution to this problem is to not change huge quantities
- of data in one transaction.
-
- If you need to spread a transaction out of lots of data, however,
- you can use subtransactions. Subtransactions allow you to manage
- Zope's memory usage yourself, so as to avoid swapping during large
- transactions.
-
- % Anonymous User - Nov. 10, 2001 7:37 am - That first sentence doesn't make sense. How about "However, if you need commit a transaction containing a lot of data you can use subtransactions." -- ChrisW
-
- Subtransactions allow you to make huge transactions. Rather than
- being limited by available memory, you are limited by available
- disk space. Each subtransaction commit writes the current changes
- out to disk and frees memory to make room for more changes.
-
- To commit a subtransaction, you first need to get a hold of a
- transaction object. Zope adds a function to get the transaction
- objects in your global namespace, 'get_transaction', and then call
- 'commit(1)' on the transaction::
-
- get_transaction().commit(1)
-
- You must balance speed, memory, and temporary storage concerns
- when deciding how frequently to commit subtransactions. The more
- subtransactions, the less memory used, the slower the operation,
- and the more temporary space used. Here's and example of how you
- might use subtransactions in your Zope code::
-
- tasks_per_subtransaction = 10
- i = 0
- for task in tasks:
- process(task)
- i = i + 1
- if i % tasks_per_subtransaction == 0:
- get_transaction().commit(1)
-
- % Anonymous User - Jan. 17, 2002 11:22 am - no "," before "and"
-
- This example shows how to commit a subtransaction at regular
- intervals while processing a number of tasks.
-
- Threads and Conflict Errors
-
- Zope is a multi-threaded server. This means that many different
- clients may be executing your Python code in different threads.
- For most cases, this is not an issue and you don't need to worry
- about it, but there are a few cases you should look out for.
-
- The first case involves threads making lots of changes to objects
- and writing to the database. The way ZODB and threading works is
- that each thread that uses the database gets its own *connection*
- to the database. Each connection gets its own *copy* of your
- object. All of the threads can read and change any of the
- objects. ZODB keeps all of these objects synchronized between the
- threads. The upshot is that you don't have to do any locking or
- thread synchronization yourself. Your code can act as through it
- is single threaded.
-
- % Anonymous User - Aug. 7, 2002 8:44 am:
- "through".replace("r", "")
-
- However, synchronization problems can occur when objects are
- changed by two different threads at the same time.
-
- Imagine that thread 1 gets its own copy of object A, as does thread
- 2. If thread 1 changes its copy of A, then thread 2 will not see
- those changes until thread 1 commits them. In cases where lots of
- objects are changing, this can cause thread 1 and 2 to try and
- commit changes to object 1 at the same time.
-
- When this happens, ZODB lets one transaction do the commit (it
- "wins") and raises a 'ConflictError' in the other thread (which
- "looses"). The looser can elect to try again, but this may raise
- yet another 'ConflictError' if many threads are trying to change
- object A. Zope does all of its own transaction management and
- will retry a losing transaction three times before giving up
- and raising the 'ConflictError' all the way up to the user.
-
- % mcdonc - Oct. 16, 2001 9:37 am - This is "loses" and "loser". No pun intended.
-
- Resolving Conflicts
-
- If a conflict happens, you have two choices. The first choice is
- that you live with the error and you try again. Statistically,
- conflicts are going to happen, but only in situations where objects
- are "hot-spots". Most problems like this can be "designed away";
- if you can redesign your application so that the changes get
- spread around to many different objects then you can usually get
- rid of the hot spot.
-
- % Anonymous User - Sep. 12, 2002 7:09 pm:
- This is totally retarded. Why can't we get a mutex instead?
-
- % Anonymous User - Sep. 12, 2002 7:18 pm:
- Code talks. Write code or shut the hell up.
-
- % Anonymous User - Sep. 13, 2002 11:20 am:
- Although the comment above was not very nice, this might be a good place
- to explain why a mutex is not appropriate.
- A mutex would have to span multiple ZEO servers and would
- serialize all Zope application threads. Conflicts are one of the prices
- of scalability, but in practice conflicts are rare enough that
- few applications need to deal with them directly.
-
- Your second choice is to try and *resolve* the conflict. In many
- situations, this can be done. For example, consider the following
- persistent object::
-
- class Counter(Persistent):
-
- self.count = 0
-
- def hit(self):
- self.count = self.count + 1
-
- This is a simple counter. If you hit this counter with a lot of
- requests though, it will cause conflict errors as different threads
- try to change the count attribute simultaneously.
-
- But resolving the conflict between conflicting threads in this
- case is easy. Both threads want to increment the self.count
- attribute by a value, so the resolution is to increment the
- attribute by the sum of the two values and make both commits
- happy; no 'ConflictError' is raised.
-
- % Anonymous User - Nov. 16, 2002 3:52 am:
- I'm with the 'code talks' aphorism - so how about a recoded example just here?
-
- To resolve a conflict, a class should define an
- '_p_resolveConflict' method. This method takes three arguments.
-
- 'oldState' -- The state of the object that the changes made by
- the current transaction were based on. The method is permitted
- to modify this value.
-
- 'savedState' -- The state of the object that is currently
- stored in the database. This state was written after 'oldState'
- and reflects changes made by a transaction that committed
- before the current transaction. The method is permitted to
- modify this value.
-
- 'newState' -- The state after changes made by the current
- transaction. The method is *not* permitted to modify this
- value. This method should compute a new state by merging
- changes reflected in 'savedState' and 'newState', relative to
- 'oldState'.
-
- % Anonymous User - Nov. 16, 2002 3:55 am:
- Does it return anything useful - I'm assuming it does (the new state, perhaps) as newstate (below) is
- described as not valid for changing. But what's the actuality?
-
- % Jace - Nov. 15, 2002 3:20 pm:
- s/define an/define a/
-
- The method should return the state of the object after resolving
- the differences.
-
- Here is an example of a '_p_resolveConflict' in the 'Counter'
- class::
-
- class Counter(Persistent):
-
- self.count = 0
-
- def hit(self):
- self.count = self.count + 1
-
- def _p_resolveConflict(self, oldState, savedState, newState):
-
- # Figure out how each state is different:
- savedDiff= savedState['count'] - oldState['count']
- newDiff= newState['count']- oldState['count']
-
- # Apply both sets of changes to old state:
- oldState['count'] = oldState['count'] + savedDiff + newDiff
-
- return oldState
-
- In the above example, '_p_resolveConflict' resolves the difference
- between the two conflicting transactions.
-
- % mcdonc - Oct. 16, 2001 9:40 am - Note that a flag on your object can be set in the form of a method named "_p_independent", which if it returns true, will prevent "read conflicts" (not talked about in this chapter, but possible) from affecting your object. Read conflicts occur in the connection's setstate method.
-
- % Anonymous User - Nov. 1, 2001 7:40 am - Someone asked about read conflicts and I needed to admit that I had no clue under what circumstance they were actually raised, but Toby Dickenson came to the rescue. The rest of the comment comes from his description of read conflicts. Zope loads objects from their persistent store into memory when they are first accessed. So if you have a transaction that touches several objects, you will be loading those objects gradually throughout the transaction. (unless they are already in memory anyway; lets assume they are not) Consider one of the last objects to be loaded..... the question is: what happens if that object has been modified in a different thread since the start of this current transaction? Ideally, Zope would load the original state of the object. That is, the state that was current at the time when this transaction started. However, this capability has not yet been developed. When it does read con
flicts will not happen. Another option would be to load the new state of that object. This is what the first revision of ZODB did, and it is bad because it leads to inconsistencies. Our transaction sees some, but not all of the changes made in a different transaction. There are some objects for which this isnt a problem; I think the LowConflictConnection class that Chris referred to can allow you to exploit this.
-
- % wilm - Nov. 15, 2001 6:59 pm - What happens if *more than two* threads try to write concurrently? Will _p_resolveConflict be called multiple times?
-
- % Anonymous User - Nov. 27, 2001 8:44 pm - (chrism, too lazy to log in) -- yes... the conflict resolution machinery will be called as many times as necessary.
-
- % Anonymous User - Dec. 29, 2001 6:39 pm - Is it neccessary to calculate the difference like this? Couldn't you just as easily used:: return oldState['count'] = newState['count'] + 1 to get the same result?
-
- % lettuce - Dec. 29, 2001 6:45 pm - Err, my bad. I meant oldState['count'] = savedState['count'] + 1; return oldState
-
- % chrism - Jan. 15, 2002 8:19 pm - Nope, what you're trying to do is to count *both* "saved" and "new" states changes against "old". If you just count the "saved" state against oldstate, you will have lost the increment caused by "new". It bugs me that this example refers to "saved" and "new" states because there really is no semantic difference to them. It should really refer to just "state1" and "state2", that might make this conceptually easier.
-
- Threadsafety of Non-Persistent Objects
-
- ZODB takes care of threadsafety for persistent objects. However,
- you must handle threadsafey yourself for non-persistent objects
- which are shared between threads.
-
- % Anonymous User - Feb. 18, 2005 5:47 pm:
- s/safey/safety/
-
- Mutable Default Arguments
-
- One tricky type of non-persistent, shared objects are mutable
- default arguments to functions, and methods. Default arguments
- are useful because they are cached for speed, and do not need to
- be recreated every time the method is called. But if these cached
- default arguments are mutable, one thread may change (mutate) the
- object when another thread is using it, and that can be bad. So,
- code like::
-
- def foo(bar=[]):
- bar.append('something')
-
- % Anonymous User - Sep. 6, 2002 9:49 am:
- need more explanation, please: what if foo is a method of class FooClass(Persistence.Persistent)? Does the
- persistence machinery guarantee threadsafety for argument 'bar' or we are in trouble all the same?
- Put another way, arguments of methods in persistent classes are persistent or not?
-
- % beyond - Oct. 5, 2002 10:55 am:
- (slightly off-topic)
- def foo(bar=[]) looks dangerous to me.
- When you use the default argument bar it will always be the same mutable
- sequence. So calling
- def myFoo(bar=[]):
- bar.append('bla')
- print bar
- two times will result in ['bla'] and ['bla','bla']
-
- Could get in trouble if two threads execute this code because lists are
- mutable. There are two solutions to this problem:
-
- o Don't use mutable default arguments. (Good)
-
- o If you use them, you cannot change them. If you want to change
- them, you will need to implement your own locking. (Bad)
-
- We recommend the first solution because mutable default arguments
- are confusing, generally a bad idea in the first place.
-
- Shared Module Data
-
- Objects stored in modules but not in the ZODB are not persistent
- and not-thread safe. In general it's not a good idea to store
- data (as opposed to functions, and class definitions) in modules
- when using ZODB.
-
- % reiman - Aug. 16, 2002 12:15 pm:
- We should mention that module data is the easiest way to achive server-lifetime data store. This is where you
- would normally store external references (file handles or database connections or session data) that you
- cannot easily reconstruct.
-
- If you decide to use module data which can change you'll need to
- protect it with a lock to ensure that only one thread at a time
- can make changes.
-
- % zigg - Jan. 16, 2002 9:31 am - See also http://www.zopelabs.com/cookbook/1006189713, which demos how to use ThreadLock. Apparently this is the "official" way?
-
- % mcdonc - Jan. 17, 2002 8:27 am - chrism - ThreadLock provides recursive locks so that the thread that holds the mutex can re-lock the lock (maybe by recursing or looping unintentionally or even intentionally) without fear of deadlock. threading.Lock does not allow this to happen.
-
- For example::
-
- from threading import Lock
- queue=[]
- l=Lock()
-
- def put(obj):
- l.acquire()
- try:
- queue.append(obj)
- finally:
- l.release()
-
- def get():
- l.acquire()
- try:
- return queue.pop()
- finally:
- l.release()
-
- Note, in most cases where you are tempted to use shared module
- data, you can likely achieve the same result with a single
- persistent object. For example, the above queue could be
- replaced with a single instance of this class::
-
- class Queue(Persistent):
-
- def __init__(self):
- self.list=[]
-
- def put(self, obj):
- self.list=self.list + [obj]
-
- def get(self):
- obj=self.list[-1]
- self.list=self.list[0:-1]
- return obj
-
- Notice how this class uses the mutable object 'self.list'
- immutably. If this class used 'self.list.pop' and
- 'self.list.append', then the persistence machinary would not
- notice that 'self.list' had changed.
-
- Shared External Resources
-
- A final category of data for which you'll need to handle
- thread-safety is external resources such as files in the
- filesystem, and other processes. In practice, these concerns
- rarely come up.
-
- Other ZODB Resources
-
- This chapter has only covered the most important features of ZODB
- from a Zope developer's perspective. Check out some of these
- sources for more in depth information:
-
- * Andrew Kuchling's "ZODB pages":http://www.kuchling.com/zodb/
- include lots of information included a programmer's guide and
- links to ZODB mailing lists.
-
- * "ZODB Wiki":http://www.zope.org/Wikis/ZODB/FrontPage has
- information about current ZODB projects.
-
- * "ZODB UML
- Model":http://www.zope.org/Documentation/Developer/Models/ZODB
- has the nitty gritty details on ZODB.
-
- * Paper "Introduction to the Zope Object
- Database":http://www.python.org/workshops/2000-01/proceedings/papers/fulton/zodb3.html
- by Jim Fulton, presented at the 8th Python Conference.
-
- Summary
-
- The ZODB is a complex and powerful system. However using
- persistent objects is almost completely painless. Seldom do you
- need to concern yourself with thread safety, transactions,
- conflicts, memory management, and database replication. ZODB takes
- care of these things for you. By following a few simple rules you
- can create persistent objects that just work.
-
- % Anonymous User - Aug. 21, 2002 12:42 pm:
- Andrew Kuchling's ZODB pages link is broken!!
-
- % Anonymous User - Aug. 21, 2002 12:53 pm:
- use http://www.amk.ca/zodb/ instead
-
- % Anonymous User - Oct. 17, 2003 6:47 am:
- The link above is broken as well!
- You can find Kuchling's guide at
- http://www.zope.org/Wikis/ZODB/guide/zodb.html
Copied: zdgbook/trunk/source/ZODBPersistentComponents.rst (from rev 96617, zdgbook/trunk/ZODBPersistentComponents.stx)
===================================================================
--- zdgbook/trunk/source/ZODBPersistentComponents.rst (rev 0)
+++ zdgbook/trunk/source/ZODBPersistentComponents.rst 2009-02-17 16:26:51 UTC (rev 96643)
@@ -0,0 +1,596 @@
+##########################
+ZODB Persistent Components
+##########################
+
+Most Zope components live in the Zope Object DataBase (ZODB).
+Components that are stored in ZODB are said to be *persistent*.
+Creating persistent components is, for the most part, a trivial
+exercise, but ZODB does impose a few rules that persistent components
+must obey in order to work properly. This chapter describes the
+persistence model and the interfaces that persistent objects can use
+to live inside the ZODB.
+
+Persistent Objects
+==================
+
+Persistent objects are Python objects that live for a long time. Most
+objects are created when a program is run and die when the program
+finishes. Persistent objects are not destroyed when the program ends,
+they are saved in a database.
+
+A great benefit of persistent objects is their transparency. As a
+developer, you do not need to think about loading and unloading the
+state of the object from memory. Zope's persistent machinery handles
+all of that for you.
+
+This is also a great benefit for application designers; you do not
+need to create your own kind of "data format" that gets saved to a
+file and reloaded again when your program stops and starts. Zope's
+persistence machinery works with *any* kind of Python objects (within
+the bounds of a few simple rules) and as your types of objects grow,
+your database simply grows transparently with it.
+
+Persistence Example
+-------------------
+
+Here is a simple example of using ZODB outside of Zope. If all you
+plan on doing is using persistent objects with Zope, you can skip
+this section if you wish.
+
+The first thing you need to do to start working with ZODB is to
+create a "root object". This process involves first opening a
+"storage" , which is the actual backend storage location for your
+data.
+
+ZODB supports many pluggable storage back-ends, but for the purposes
+of this article we're going to show you how to use the 'FileStorage'
+back-end storage, which stores your object data in a file. Other
+storages include storing objects in relational databases, Berkeley
+databases, and a client to server storage that stores objects on a
+remote storage server.
+
+
+To set up a ZODB, you must first install it. ZODB comes with Zope,
+so the easiest way to install ZODB is to install Zope and use the
+ZODB that comes with your Zope installation. For those of you who
+don't want all of Zope, but just ZODB, see the instructions for
+downloading ZODB from the `ZODB web page
+<http://wiki.zope.org/ZODB>`_.
+
+After installing ZODB, you can start to experiment with it right from
+the Python command line interpreter. If you've installed Zope,
+before running this set of commands, shut down your Zope server, and
+"cd" to the "lib/python" directory of your Zope instance. If you're
+using a "standalone" version of ZODB, you likely don't need to do
+this, and you'll be able to use ZODB by importing it from a standard
+Python package directory. In either case, try the following set of
+commands::
+
+ chrism at saints:/opt/zope/lib/python$ python
+ Python 2.1.1 (#1, Aug 8 2001, 21:17:50)
+ [GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2
+ Type "copyright", "credits" or "license" for more information.
+ >>> from ZODB import FileStorage, DB
+ >>> storage = FileStorage.FileStorage('mydatabase.fs')
+ >>> db = DB( storage )
+ >>> connection = db.open()
+ >>> root = connection.root()
+
+Here, you create storage and use the 'mydatabse.fs' file to store the
+object information. Then, you create a database that uses that
+storage.
+
+
+Next, the database needs to be "opened" by calling the 'open()'
+method. This will return a connection object to the database. The
+connection object then gives you access to the 'root' of the database
+with the 'root()' method.
+
+The 'root' object is the dictionary that holds all of your persistent
+objects. For example, you can store a simple list of strings in the
+root object::
+
+ root['employees'] = ['Bob', 'Mary', 'Jo']
+
+Now, you have changed the persistent database by adding a new object,
+but this change is so far only temporary. In order to make the
+change permanent, you must commit the current transaction::
+
+ get_transaction().commit()
+
+Transactions are ways to make a lot of changes in one atomic
+operation. In a later article, we'll show you how this is a very
+powerful feature. For now, you can think of committing transactions
+as "checkpoints" where you save the changes you've made to your
+objects so far. Later on, we'll show you how to abort those changes,
+and how to undo them after they are committed.
+
+If you had used a relational database, you would have had to issue a
+SQL query to save even a simple python list like the above example.
+You would have also needed some code to convert a SQL query back into
+the list when you wanted to use it again. You don't have to do any
+of this work when using ZODB. Using ZODB is almost completely
+transparent, in fact, ZODB based programs often look suspiciously
+simple!
+
+Working with simple python types is useful, but the real power of
+ZODB comes out when you store your own kinds of objects in the
+database. For example, consider a class that represents a employee::
+
+ from Persistence import Persistent
+
+ class Employee(Persistent):
+
+ def setName(self, name):
+ self.name = name
+
+
+Calling 'setName' will set a name for the employee. Now, you can put
+Employee objects in your database::
+
+ for name in ['Bob', 'Mary', 'Joe']:
+ employee = Employee()
+ employee.setName(name)
+ root['employees'].append(employee)
+
+ get_transaction().commit()
+
+Don't forget to call 'commit()', so that the changes you have made so
+far are committed to the database, and a new transaction is begun.
+
+Persistent Rules
+================
+
+There are a few rules that must be followed when your objects are
+persistent.
+
+- Your objects, and their attributes, must be "pickleable".
+
+- Your object cannot have any attributes that begin with '_p_'.
+
+- Attributes of your object that begin with '_v_' are "volatile" and
+ are not saved to the database (see next section).
+
+- You must explicitly signal any changes made to mutable attributes
+ (such as instances, lists, and dictionaries) or use persistent
+ versions of mutable objects, like 'ZODB.PersistentMapping' (see
+ below for more information on 'PersistentMapping'.)
+
+In this section, we'll look at each of these special rules one by
+one.
+
+The first rules says that your objects must be pickleable. This
+means that they can be serialized into a data format with the
+"pickle" module. Most python data types (numbers, lists,
+dictionaries) can be pickled. Code objects (method, functions,
+classes) and file objects (files, sockets) *cannot* be pickled.
+Instances can be persistent objects if:
+
+- They subclass 'Persistence.Persistent'
+
+- All of their attributes are pickleable
+
+The second rule is that none of your objects attributes can begin
+with '_p_'. For example, '_p_b_and_j' would be an illegal object
+attribute. This is because the persistence machinery reserves all of
+these names for its own purposes.
+
+The third rule is that all object attributes that begin with '_v_'
+are "volatile" and are not saved to the database. This means that as
+long as the persistent object is in Zope memory cache, volatile
+attributes can be used. When the object is deactivated (removed from
+memory) volatile attributes are thrown away.
+
+Volatile attributes are useful for data that is good to cache for a
+while but can often be thrown away and easily recreated. File
+connections, cached calculations, rendered templates, all of these
+kinds of things are useful applications of volatile attributes. You
+must exercise care when using volatile attributes. Since you have
+little control over when your objects are moved in and out of memory,
+you never know when your volatile attributes may disappear.
+
+The fourth rule is that you must signal changes to mutable types.
+This is because persistent objects can't detect when mutable types
+change, and therefore, doesn't know whether or not to save the
+persistent object or not.
+
+For example, say you had a list of names as an attribute of your
+object called 'departments' that you changed in a method called
+'addDepartment'::
+
+ class DepartmentManager(Persistent):
+
+ def __init__(self):
+ self.departments = []
+
+ def addDepartment(self, department):
+ self.departments.append(department)
+
+When you call the 'addDepartment' method you change a mutable type,
+'departments' but your persistent object will not save that change.
+
+There are two solutions to this problem. First, you can assign a
+special flag, '_p_changed'::
+
+ def addDepartment(self, department):
+ self.department.append(department)
+ self._p_changed = 1
+
+Remember, '_p_' attributes do something special to the persistence
+machinery and are reserved names. Assigning 1 to '_p_changed' tells
+the persistence machinery that you changed the object, and that it
+should be saved.
+
+Another technique is to use the mutable attribute as though it were
+immutable. In other words, after you make changes to a mutable
+object, reassign it::
+
+ def addDepartment(self, department):
+ departments = self.departments
+ departments.append(department)
+ self.department = departments
+
+Here, the 'self.departments' attribute was re-assigned at the end of
+the function to the "working copy" object 'departments'. This
+technique is cleaner because it doesn't have any explicit
+'_p_changed' settings in it, but this implicit triggering of the
+persistence machinery should always be understood, otherwise use the
+explicit syntax.
+
+A final option is to use persistence-aware mutable attributes such as
+'PersistentMapping', and 'IOBTree'. 'PersistentMapping' is a mapping
+class that notifies ZODB when you change the mapping. You can use
+instances of 'PersistentMapping' in place of standard Python
+dictionaries and not worry about signaling change by reassigning the
+attribute or using '_p_changed'. Zope's Btree classes are also
+persistent-aware mutable containers. This solution can be cleaner
+than using mutable objects immutably, or signaling change manually
+assuming that there is a persistence-aware class available that meets
+your needs.
+
+Transactions and Persistent Objects
+===================================
+
+When changes are saved to ZODB, they are saved in a *transaction*.
+This means that either all changes are saved, or none are saved. The
+reason for this is data consistency. Imagine the following scenario:
+
+1. A user makes a credit card purchase at the sandwich.com website.
+
+2. The bank debits their account.
+
+3. An electronic payment is made to sandwich.com.
+
+Now imagine that an error happens during the last step of this
+process, sending the payment to sandwich.com. Without transactions,
+this means that the account was debited, but the payment never went
+to sandwich.com! Obviously this is a bad situation. A better
+solution is to make all changes in a transaction:
+
+1. A user makes a credit card purchase at the sandwich.com website.
+
+2. The transaction begins
+
+3. The bank debits their account.
+
+4. An electronic payment is made to sandwich.com.
+
+5. The transaction commits
+
+Now, if an error is raised anywhere between steps 2 and 5, *all*
+changes made are thrown away, so if the payment fails to go to
+sandwich.com, the account won't be debited, and if debiting the
+account raises an error, the payment won't be made to sandwich.com,
+so your data is always consistent.
+
+When using your persistent objects with Zope, Zope will automatically
+*begin* a transaction when a web request is made, and *commit* the
+transaction when the request is finished. If an error occurs at any
+time during that request, then the transaction is *aborted*, meaning
+all the changes made are thrown away.
+
+If you want to *intentionally* abort a transaction in the middle of a
+request, then just raise an error at any time. For example, this
+snippet of Python will raise an error and cause the transaction to
+abort::
+
+ raise SandwichError('Not enough peanut butter.')
+
+A more likely scenario is that your code will raise an exception when
+a problem arises. The great thing about transactions is that you
+don't have to include cleanup code to catch exceptions and undo
+everything you've done up to that point. Since the transaction is
+aborted the changes made in the transaction will not be saved.
+
+Because Zope does transaction management for you, most of the time
+you do not need to explicitly begin, commit or abort your own
+transactions. For more information on doing transaction management
+manually, see the links at the end of this chapter that lead to more
+detailed tutorials of doing your own ZODB programming.
+
+
+Subtransactions
+---------------
+
+Zope waits until the transaction is committed to save all the changes
+to your objects. This means that the changes are saved in memory.
+If you try to change more objects than you have memory in your
+computer, your computer will begin to swap and thrash, and maybe even
+run you out of memory completely. This is bad. The easiest solution
+to this problem is to not change huge quantities of data in one
+transaction.
+
+If you need to spread a transaction out of lots of data, however, you
+can use subtransactions. Subtransactions allow you to manage Zope's
+memory usage yourself, so as to avoid swapping during large
+transactions.
+
+Subtransactions allow you to make huge transactions. Rather than
+being limited by available memory, you are limited by available disk
+space. Each subtransaction commit writes the current changes out to
+disk and frees memory to make room for more changes.
+
+To commit a subtransaction, you first need to get a hold of a
+transaction object. Zope adds a function to get the transaction
+objects in your global namespace, 'get_transaction', and then call
+'commit(1)' on the transaction::
+
+ get_transaction().commit(1)
+
+You must balance speed, memory, and temporary storage concerns when
+deciding how frequently to commit subtransactions. The more
+subtransactions, the less memory used, the slower the operation, and
+the more temporary space used. Here's and example of how you might
+use subtransactions in your Zope code::
+
+ tasks_per_subtransaction = 10
+ i = 0
+ for task in tasks:
+ process(task)
+ i = i + 1
+ if i % tasks_per_subtransaction == 0:
+ get_transaction().commit(1)
+
+This example shows how to commit a subtransaction at regular
+intervals while processing a number of tasks.
+
+Threads and Conflict Errors
+---------------------------
+
+Zope is a multi-threaded server. This means that many different
+clients may be executing your Python code in different threads. For
+most cases, this is not an issue and you don't need to worry about
+it, but there are a few cases you should look out for.
+
+The first case involves threads making lots of changes to objects and
+writing to the database. The way ZODB and threading works is that
+each thread that uses the database gets its own *connection* to the
+database. Each connection gets its own *copy* of your object. All
+of the threads can read and change any of the objects. ZODB keeps
+all of these objects synchronized between the threads. The upshot is
+that you don't have to do any locking or thread synchronization
+yourself. Your code can act as though it is single threaded.
+
+However, synchronization problems can occur when objects are changed
+by two different threads at the same time.
+
+Imagine that thread 1 gets its own copy of object A, as does thread
+2. If thread 1 changes its copy of A, then thread 2 will not see
+those changes until thread 1 commits them. In cases where lots of
+objects are changing, this can cause thread 1 and 2 to try and commit
+changes to object 1 at the same time.
+
+When this happens, ZODB lets one transaction do the commit (it
+"wins") and raises a 'ConflictError' in the other thread (which
+"looses"). The looser can elect to try again, but this may raise yet
+another 'ConflictError' if many threads are trying to change object
+A. Zope does all of its own transaction management and will retry a
+losing transaction three times before giving up and raising the
+'ConflictError' all the way up to the user.
+
+
+Resolving Conflicts
+-------------------
+
+If a conflict happens, you have two choices. The first choice is that
+you live with the error and you try again. Statistically, conflicts
+are going to happen, but only in situations where objects are
+"hot-spots". Most problems like this can be "designed away"; if you
+can redesign your application so that the changes get spread around
+to many different objects then you can usually get rid of the hot
+spot.
+
+
+Your second choice is to try and *resolve* the conflict. In many
+situations, this can be done. For example, consider the following
+persistent object::
+
+ class Counter(Persistent):
+
+ self.count = 0
+
+ def hit(self):
+ self.count = self.count + 1
+
+This is a simple counter. If you hit this counter with a lot of
+requests though, it will cause conflict errors as different threads
+try to change the count attribute simultaneously.
+
+But resolving the conflict between conflicting threads in this case
+is easy. Both threads want to increment the self.count attribute by
+a value, so the resolution is to increment the attribute by the sum
+of the two values and make both commits happy; no 'ConflictError' is
+raised.
+
+
+To resolve a conflict, a class should define an '_p_resolveConflict'
+method. This method takes three arguments.
+
+'oldState' -- The state of the object that the changes made by the
+current transaction were based on. The method is permitted to modify
+this value.
+
+'savedState' -- The state of the object that is currently stored in
+the database. This state was written after 'oldState' and reflects
+changes made by a transaction that committed before the current
+transaction. The method is permitted to modify this value.
+
+'newState' -- The state after changes made by the current
+transaction. The method is *not* permitted to modify this
+value. This method should compute a new state by merging changes
+reflected in 'savedState' and 'newState', relative to 'oldState'.
+
+The method should return the state of the object after resolving the
+differences.
+
+Here is an example of a '_p_resolveConflict' in the 'Counter' class::
+
+ class Counter(Persistent):
+
+ self.count = 0
+
+ def hit(self):
+ self.count = self.count + 1
+
+ def _p_resolveConflict(self, oldState, savedState, newState):
+
+ # Figure out how each state is different:
+ savedDiff= savedState['count'] - oldState['count']
+ newDiff= newState['count']- oldState['count']
+
+ # Apply both sets of changes to old state:
+ oldState['count'] = oldState['count'] + savedDiff + newDiff
+
+ return oldState
+
+In the above example, '_p_resolveConflict' resolves the difference
+between the two conflicting transactions.
+
+Threadsafety of Non-Persistent Objects
+======================================
+
+ZODB takes care of threadsafety for persistent objects. However, you
+must handle threadsafey yourself for non-persistent objects which are
+shared between threads.
+
+Mutable Default Arguments
+-------------------------
+
+One tricky type of non-persistent, shared objects are mutable default
+arguments to functions, and methods. Default arguments are useful
+because they are cached for speed, and do not need to be recreated
+every time the method is called. But if these cached default
+arguments are mutable, one thread may change (mutate) the object when
+another thread is using it, and that can be bad. So, code like::
+
+ def foo(bar=[]):
+ bar.append('something')
+
+
+Could get in trouble if two threads execute this code because lists
+are mutable. There are two solutions to this problem:
+
+- Don't use mutable default arguments. (Good)
+
+- If you use them, you cannot change them. If you want to change
+ them, you will need to implement your own locking. (Bad)
+
+We recommend the first solution because mutable default arguments are
+confusing, generally a bad idea in the first place.
+
+Shared Module Data
+------------------
+
+Objects stored in modules but not in the ZODB are not persistent and
+not-thread safe. In general it's not a good idea to store data (as
+opposed to functions, and class definitions) in modules when using
+ZODB.
+
+
+If you decide to use module data which can change you'll need to
+protect it with a lock to ensure that only one thread at a time can
+make changes.
+
+
+For example::
+
+ from threading import Lock
+ queue=[]
+ l=Lock()
+
+ def put(obj):
+ l.acquire()
+ try:
+ queue.append(obj)
+ finally:
+ l.release()
+
+ def get():
+ l.acquire()
+ try:
+ return queue.pop()
+ finally:
+ l.release()
+
+Note, in most cases where you are tempted to use shared module data,
+you can likely achieve the same result with a single persistent
+object. For example, the above queue could be replaced with a single
+instance of this class::
+
+ class Queue(Persistent):
+
+ def __init__(self):
+ self.list=[]
+
+ def put(self, obj):
+ self.list=self.list + [obj]
+
+ def get(self):
+ obj=self.list[-1]
+ self.list=self.list[0:-1]
+ return obj
+
+Notice how this class uses the mutable object 'self.list'
+immutably. If this class used 'self.list.pop' and 'self.list.append',
+then the persistence machinary would not notice that 'self.list' had
+changed.
+
+Shared External Resources
+=========================
+
+A final category of data for which you'll need to handle
+thread-safety is external resources such as files in the filesystem,
+and other processes. In practice, these concerns rarely come up.
+
+Other ZODB Resources
+====================
+
+This chapter has only covered the most important features of ZODB
+from a Zope developer's perspective. Check out some of these sources
+for more in depth information:
+
+- Andrew Kuchling's `ZODB pages <http://www.kuchling.com/zodb/>`_
+ include lots of information included a programmer's guide and links
+ to ZODB mailing lists.
+
+- `ZODB Wiki <http://wiki.zope.org/ZODB>`_ has information about
+ current ZODB projects.
+
+- `ZODB UML
+ Model <http://www.zope.org/Documentation/Developer/Models/ZODB>`_ has
+ the nitty gritty details on ZODB.
+
+- Paper `Introduction to the Zope Object Database
+ <http://www.python.org/workshops/2000-01/proceedings/papers/fulton/zodb3.html>`_
+ by Jim Fulton, presented at the 8th Python Conference.
+
+Summary
+=======
+
+The ZODB is a complex and powerful system. However using persistent
+objects is almost completely painless. Seldom do you need to concern
+yourself with thread safety, transactions, conflicts, memory
+management, and database replication. ZODB takes care of these things
+for you. By following a few simple rules you can create persistent
+objects that just work.
+
Property changes on: zdgbook/trunk/source/ZODBPersistentComponents.rst
___________________________________________________________________
Added: svn:mergeinfo
+
Modified: zdgbook/trunk/source/index.rst
===================================================================
--- zdgbook/trunk/source/index.rst 2009-02-17 13:16:30 UTC (rev 96642)
+++ zdgbook/trunk/source/index.rst 2009-02-17 16:26:51 UTC (rev 96643)
@@ -14,7 +14,8 @@
Introduction.rst
ComponentsAndInterfaces.rst
ObjectPublishing.rst
- Products.rst
+ Products.rst
+ ZODBPersistentComponents.rst
AppendixA.rst
AppendixB.rst
More information about the Checkins
mailing list