[ZODB-Dev] Blobs revisited...

[Chris McDonough]

On Oct 5, 2005, at 6:11 AM, Jim Fulton wrote:

> Chris McDonough wrote:
>
>> FYI,
>> Last week, Christian Theune and I muscled the elderly ctheune-  
>> blobsupport-branch of ZODB into shape to work again against the  
>> ZODB  trunk.
>>
>
> Yay!  Did you also resolve the code issues I pointed out
> to Christian?  The original "workingness" was a bit suspect. :)

The one issue I knew about was that _abort was never called on a  
BlobStorage when a transaction is aborted... it couldn't have been  
because there was basically a bit of nonsensical code in the method  
that would have raised an exception.

I don't think we solved this.  I think we did figure out why it  
wasn't called -- it seems that methods of a zope.proxy-wrapped object  
aren't rebound to the wrapper but instead to the wrapped object.  I  
then tried every which way to have some cleanup code invoked on abort  
(overriding "tpc_abort" instead of "_abort", overriding "abort"  
instead of "_abort") but without success.  I think I'm doing  
something stupid.  Late in the day I think we figured that the abort  
cleanup code might need to go in the "BlobDataManager" (each blob has  
its own data manager) instead of the BlobStorage because it's unclear  
when BlobStorage's tpc_abort/_abort/abort methods will be called.

In short, confusion reigns. ;-)

>
>> We ran into some "interesting" issues with supporting savepoints   
>> (difficult to do efficiently with blobs, so we didn't try), I  
>> added  some somewhat suspect code to support blobs using a  
>> TmpStore, and we  as always need to write some more tests, but it  
>> works and the result  is on the blob-merge-branch of ZODB in SVN.
>>
>
> Could you eleborate or point at a document that does?

Sure.  BlobStorage is a wrapper around an existing storage.  The  
wrapped storage's connections might support savepoints, which implies  
that they support rollback.  But supporting rollback for a  
BlobDataManager is non-trivial because you'd need to make copies of  
all writable blobs involved in the transaction and maintain the file  
position state for all writeable and readable blobs in order to roll  
back to exactly that state.  This was more than we could chew off  
during the sprint, although it'd be possible.  But essentially right  
now, if you have a transaction that involves any blobs and you want  
to use a savepoint (other than an "optimistic" one), it will raise an  
error.

I hacked TmpStore to have loadBlob and storeBlob methods (required  
because during a savepoint, TmpStore "becomes" the storage) but there  
is currently no code to clean up the temporary blob files created  
while TmpStore is in use.  TmpStore is also currently very simple and  
does no mutex locking like other storages.  I don't really understand  
why not, so this bugs me.

>> We also created a Zope branch to handle the ZODB head.  Namely,  
>> we  used multidatabase support.  This is in the zodb-blobs-branch  
>> of Zope  in SVN.  This basically consisted of ripping out old  
>> DBTab code and  replacing it with calls into ZODB's multidatabase  
>> support.  It also  works.
>>
>
> Yay.  Does this actually have anything to do with blobs?

Nope.  We just needed to solve the mounting problem in order to test  
the ZODB blob-merge-branch within Zope.

> I would rather have seen these two projects remain separate.
>
> Is there a Bloc-specific ZODB branch now? Or is all of the work in a
> Zope branch? (I'm a bit confused here.)

There is a ZODB branch named "blob-merge-branch".  That's where all  
the work is and you can use this without worrying about Zope.

There is a Zope branch named zodb-blobs-branch that uses the ZODB  
"blobs-merge-branch" in various SVN-externals as opposed to a tagged  
ZODB version.  It also has the changes to support multidatabases.   
This is really just a convenience.

>
>> I'm still working on this in my downtime.  If anyone else is   
>> interested in helping out, let me know!
>>
>
> Thanks for the work!  I'll try to squeeze in some time soon to  
> review the current
> blob code.  Where is it?

Cool!

svn co svn+ssh://svn.zope.org/repos/main/ZODB/branches/blob-merge-branch

- C