[ZODB-Dev] Amazon SimpleDB Adapter

[David Pratt]

Hi guys, I had thought of something along these lines a while back  
for s3 and discovered laurence had already started something - but it  
was too slow which I can understand.  Can't really see things being  
much better with simpledb and I think cost would not be great when  
you start thinking of paying for all the pickles.

I think a better way of interacting with databases these days is to  
use models in a direct way rather than bend the storage to the zodb  
api. Martijn and Laurence have been doing this with megrok.rdb,  
megrok.rdf for illustration. The work to tie transactions together  
with zope is really the important stuff.

There is no lack of storage options out there these days. My opinion  
is that if you want the true characteristics of the storage, you  
won't make it behave like a zodb. Lovely systems came to a similar  
conclusion after trying relstorage and then moving to use storm in a  
more direct way.

Regards,
David


On Oct 12, 2008, at 12:50 AM, Shane Hathaway wrote:

> Laurence Rowe wrote:
>> I'm not sure RelStorage is the best place for it - SimpleDB is very
>> different to relational databases.
>
> RelStorage doesn't use much of a relational database either (except
> during packing).
>
>> A couple of years ago I experimented with s3storage [1]. This  
>> turned out
>> to be very slow due to the number of writes performed every  
>> transaction
>> - one per object, though this could be improved if the writes were
>> parallelized. It reached the point where zope2 would start up.  
>> This took
>> about 10 or 15 mintutes at the time (I did not have access to EC2  
>> at the
>> time and this was over public wifi).
>>
>> It worked by creating it's own indexes in S3. I don't think SimpleDB
>> will give any advantage unless it is shown to be faster to query than
>> S3. You cannot store pickles directly in SimpleDB because it is  
>> limited
>> to an attribute size of 1024 bytes.
>>
>> The challenge in building such a system is in Amazon's eventual
>> consistency model means you cannot know how up to date your view  
>> of the
>> data is. I think it could make a great backend for storing pickles
>> (keyed by oid, tid) but it is probably much easier to have a separate
>> index to consult during loadSerial.
>
> Thanks for the background on S3 and SimpleDB.  Using Amazon's storage
> services as a ZODB backend is sounding ever more like an interesting
> challenge.
>
> Shane
>
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev