[ZODB-Dev] [OT] NoSQL

[Roché Compaan]

On Fri, 2009-11-13 at 10:58 +0100, Christian Theune wrote:
> On 11/13/2009 10:42 AM, Adam GROSZER wrote:
> > Hello,
> > 
> > I think we can look at this at 2 levels.
> > 
> > 1.: As your app uses ZODB. Then this is your app's
> >     problem/reponsibility. You use a nosql contender directly from
> >     your app and it's your responsibility to deal with it.
> > 
> > 2.: On the ZODB Storage level. So far I can see that level needs
> >     consistency, transactions and locking support. Those are usually
> >     missing from nosql implementations (unless I miss some).
> >     OTOH a key-value store would fit the ZODB storage.
> >     If someone finds/writes a key-value storage that has the above
> >     properties we could give it a try.
> 
> Looking at the article referenced by Shane I understand that we'd have
> to drop consistency for being able to use such a store -- I feel that's
> not a good idea in the face of ZODB. The applications we write are
> intended to run consistently without having application-level (or even
> user-based) reconciliation work to do if a transaction came through.

I think there is something more important to notice than dropping of
consistency, and that is scaling easily to handle very large datasets.

Having to implement a data partitioning strategy at application level is
very difficult to get right. Having a ZODB storage that is distributed
across machines can become a big selling point for the ZODB and would
make it very convenient for us who do sometimes have the rare
opportunity to write applications that expect data sets in excess of 100
million records.

We had such an opportunity about 2 years ago and although the client
never reached (and probably will never) reach the membership they
dreamed about, they did pay us to develop a storage for members that
could scale to more than a 100 million members. We implemented a data
partitioning strategy at application level. If I had another shot at it,
I would try and develop a distributed ZODB storage, because it would be
a lot simpler compared to what we had to do at application level.

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za