[ZODB-Dev] Persisting Python objects in MongoDB

[Stephan Richter]

Hi everyone,

as you might have seen from my checkins on Friday, I released mongopersist. In 
short, it implements a persistence backend in MongoDB (that, with a little bit 
of effort, could be used as a ZODB replacement in Zope):

http://pypi.python.org/pypi/mongopersist

Some background and motivation:

Many years ago, Jim told me that the persistence code could be used to 
implement any storage backend. But until now I never had the need to confirm 
that statement. The ZODB and its various storages served me just fine. But then 
I decided to use MongoDB as storage for my latest project. (I just recently 
used storm with PostGreSQL for a project and was once again annoyed about the 
impedance mismatch between the stiff relational and flexible object worlds.)

It was out of question that I just wanted to use pymongo straight up, since it 
makes admin interfaces just very painful to write. I first tried p01.mongo, but 
unfortunately it tries to re-implement persistence with some of the most 
annoying ORM-style features (such as explicit attribute declarations).

So, I came to the conclusion I wanted to build an abstraction layer myself and 
see how well it would work. An ORM-style solution was out of question. I 
consider attribute-type declarations ORM-style, since it prohibits me to 
extend objects in arbitrary ways (think annotations, sub-classing, etc). I 
then looked into implementing a ZODB storage. Unfortunately, the ZODB really 
wants to deal with pickles and I wanted to store objects in a way that the 
data could be used without Python.

Thus, implementing a storage mechanism on top persistence seemed like the best 
choice. It was not too hard, since ZODB's storage implementations provided 
good examples. Here are some of the main features:

* Full persistent data manager implementation making storing objects seamless.

* Data manager can be used to load and dump objects directly.

* Automatic and explicit specification of the mongo collection to use.

* Store multiple object types in the same collection. This is particularly 
useful, if you have sub-classes of an object type.

* Storage of non-persistent objects. Pretty much everything that can be 
pickled can be stored. There is some work left to do to support all pickle 
hooks and object constructors.

* Plugins to provide custom serializers and de-serializers for objects. This 
is useful for objects whose reduced state is not suitable for Mongo storage, 
such as datetime.date.

* Control over document granularity. In the ZODB, any persistent object causes 
a separate pickle/document. In mongopersist you can choose whether a 
persistent object is stored as part of its parent object or not. (The Mongo 
guys make a big deal about choosing document granularity.)

* Cross database support, i.e. full DBRef object references are used 
everywhere.

* Optional write-conflict detection.

* Simple Mapping implementation that represents a collection.

* Optional Zope Support: Multiple container implementations for various use 
cases. The main container supports one collection for multiple instances by 
keeping a parent reference. The container itself can be connected to a ZODB or 
mongopersist. The container also provides full access to the MongoDB query 
API, filtering by parent automatically and supporting raw or object results.

Before you jump on this right away, here are some features that are missing:

* BTreeContainer instances will probably store, but they would not provide 
sensible output in Mongo. Custom hooks need to be written to handle 
BTreeContainers sensibly.

* Since the pickle module does not cleanly separate reducing an object and 
creating a pickle string, mongopersist must effectively re-implement the pickle 
API. I have implemented the most common use cases, but some pickle hooks and 
handling of classes with __new__ constructors are still missing.

* Cyclic references of non-persistent objects is not supported.

That said, we are about to head into production with mongopersist. If you need 
any help or want those above two problems addressed, feel free to contact me.

So was Jim right? Absolutely!

Regards,
Stephan
-- 
Entrepreneur and Software Geek
Google me. "Zope Stephan Richter"