[ZODB-Dev] Amazon SimpleDB Adapter

[Laurence Rowe]

Shane Hathaway wrote:
> Benjamin Liles wrote:
>> Currently at the Plone conference it seems that a large number of people
>> are beginning to host their Plone sites on the Amazon EC2 service.  A
>> simpleDB adapter might be a good way to provide persistent storage for
>> an EC2 base Zope instance.  Has there been any interest in this?  If I
>> was to write one, should I add it to RelStorage or create my own package
>> along the lines of relstorage.adapters.simpledb?
> 
> This sounds interesting!  We should add an adapter to RelStorage.  We
> might run into some trouble with MVCC, but I think we can solve that.
> We should also use Amazon S3 directly for blob storage.
> 
> In general, Amazon's services seem a much better fit for ZODB apps than
> what Google is offering.

I'm not sure RelStorage is the best place for it - SimpleDB is very 
different to relational databases.

A couple of years ago I experimented with s3storage [1]. This turned out 
to be very slow due to the number of writes performed every transaction 
- one per object, though this could be improved if the writes were 
parallelized. It reached the point where zope2 would start up. This took 
about 10 or 15 mintutes at the time (I did not have access to EC2 at the 
time and this was over public wifi).

It worked by creating it's own indexes in S3. I don't think SimpleDB 
will give any advantage unless it is shown to be faster to query than 
S3. You cannot store pickles directly in SimpleDB because it is limited 
to an attribute size of 1024 bytes.

The challenge in building such a system is in Amazon's eventual 
consistency model means you cannot know how up to date your view of the 
data is. I think it could make a great backend for storing pickles 
(keyed by oid, tid) but it is probably much easier to have a separate 
index to consult during loadSerial.

It may also be worth experimenting with DirectoryStorage over s3fs [2].

Laurence


[1] http://code.google.com/p/s3storage

[2] http://code.google.com/p/s3fs