[ZODB-Dev] Storing blob data out of zodb into amazon s3?

[Jim Fulton]

On Thu, Jul 5, 2012 at 3:05 PM, Noe Nieto <nnieto at noenieto.com> wrote:
> I was wondering if it's possible to write some sort of "cloudstorage" plugin
> for ZODB that saves blobs in amazon S3 or something similar.
>
> The idea is cool because all things cloud are cool.

Um, well no. :)

But if you're in AWS, I think there are some advantages to S3 over
EBS.

> But there might be some
> shortcomings like performance or something like that. What do you think?

I took an initial stab, as Satchit pointed out.  It was a bit naive.
S3 is too slow, especially when committing transactions.

I wanna make this work though.  My current thinking is:

- Save to the blob directory.

- Have a separate process that moves files to S3 (copy to s3, then
  delete from blob directory on success or later).

- On blob load, if the file can't be found on the local directory,
fetch from S3.

There are a number of possible variations:

- Don't delete from the blob directory immediately after upload to use
  it as a cache for new content.

- Have the client look in S3 before contacting the ZEO server
  (although that has the downside of having to distribute keys to the
  clients).  (Maybe only do this if the serial is more than some time
  in the past.)

- Have the client fetch from S3 if the blob isn't in the ZEO server's
  blob directory (as indicated by a response from the ZEO server).

- Have a separate process that serves the blobs deom the blob
  directory or from S3 via HTTP.

  I did some experiments that showed significant performance benefits
  in a heavily loaded ZEO server in serving blobs from an separate
  HTTP server.

If you're going to serve blob data directly, without modification, it
may make sense to store it in S3 at the application level, rather than
going through ZODB.  Alternatively, it would be nice to have a way to
get the S3 location for a blob.  Maybe this could be done through some
extension API. <shrug>

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm