[Zope-dev] Re: The remaining spanner in the works :-)

[Shane Hathaway]

Chris Withers wrote:
> Shane Hathaway wrote:
> 
> <snip>
> 
> Wow! That was very very cool. I look forward to helping make this all a 
> reality in Zope 3 ;-)
> 
>> I'm sure I left a few things out, so ask questions about the unclear
>> parts.  It's probably more info than you were expecting. ;-)
> 
> 
> Hmm, okay, the only problem I can think that still needs solving is this:
> Say you're serializing to a relational database. All well and good.
> 
> ...until another app comes along and modifies your relational table 
> (pretty common usecase, othwerwise people wouldn't care as much about 
> the storage layer...)
> 
> How does the serializer/deserializer find out something has changed and 
> propogate this back up, invalidating any cached object, etc?

I've been trying out a limited-duration cache strategy.  The simplest 
approach is to clear the cache between transactions.  Alternatively, you 
can clear the cache periodically.  For a lot of applications this is 
adequate.

Another approach, if you can afford it, is triggers.  To do this, you 
have to know a lot about your database, since there is no standard way.

But as I learned from AdaptableStorage, no matter what you do, you need 
to have conflict detection.  Otherwise Zope will merrily delete a 
directory you just added externally, for example.  This was a hard 
problem to solve until I stumbled on the idea of putting half of an MD5 
sum in _p_serial instead of the date.  (Only half because the current C 
code only allows 8 bytes in _p_serial. ;-) )  This works well enough to 
detect nearly all conflicts, even though it might not be the speediest 
solution.

And as it turned out, as long as I had conflict detection, it didn't 
matter as much that the database didn't have the most recent state all 
the time.  Zope detected conflict errors and retried the transactions, 
and the second time always worked.  It was good. :-)

Finally, here's a theoretical solution that I think would be ideal for a 
lot of people: if we could just ask the RDBMS for its current 
transaction ID, Zope could keep track of the last transaction ID it knew 
about, and clear the caches when another process made a change.  This 
solution may yield the highest performance for Zope-centric 
applications.  It would also remove the need for my MD5 hack. ;-)

Shane