[ZODB-Dev] shared cache when no write?

[Dylan Jay]

On 14/12/2012, at 8:32 AM, Jim Fulton <jim at zope.com> wrote:

> On Thu, Dec 13, 2012 at 4:18 PM, Dylan Jay <djay at pretaweb.com> wrote:
> ...
>> I'd never considered that the cache was attached to the db connection rather
>> than the thread. I just reread
>> http://docs.zope.org/zope2/zope2book/MaintainingZope.html and it says
>> exactly that.
>> So what your saying is I'd tune db connections down to memory size on an
>> instance dedicated to io bound and then increase the threads. Whenever a
>> thread requests a db connection and there isn't one available it will block.
>> So I just optimize my app the release the db connection when not needed.
>> In fact I could tune all my copes this way since a zone with 10 threads and
>> 2 connections is going to end up queuing requests the same as 2 threads and
>> 10 connections?
> 
> Something like that. It's a little more complicated than that because
> Zope 2 is managing connections for you, it would be easy to run afoul
> of that.  This is a case where something that usually makes your life
> easier, makes it harder. :)

true. With Plone as you have many modules sharing the connection all expecting it to be the same connection closing the connection half way through isn't possible. If it was closed and another connection opened then the other modules that are outside of your control might have references to stale data.

> 
> What I'd do is use a separate database other than the one Zope 2 is
> using.  Then you can manage connections yourself without conflicting
> with the publisher is doing.  Then, when you want to use the database,
> you just open the database, being careful to close it when you're
> going to block.  The downside being that you'll have separate
> transactions.
> 
>> This should be easier to achieve and changes the application less than the
>> erp5 background task solution mentioned.
> 
> It would probably be a good idea to lean more bout how erp does this.
> The erp approach sounds like a variation on what I suggested.

It's not always possible as sometimes you need to feedback the result to the user immediately. 
Let's take another example. A Plone site with a page that lets you upload a mp3 file and it guesses the song, then combines that with your preference data to return other songs you might like. The guessing the song bit is an external service and the preference data is stored in the same zodb as Plone. 
To do it the ERP background task way you;d deliver back a page with some javascript on it that polls the server to see if the song had been processed yet. This isn't always desirable, esp if you have to avoid javascript.  

Maybe another possibility is to do it the way ZODB handles streaming blobs. The blob streaming happens after the db connection is closed. Perhaps if there was a way to register a callback in zope for processing to happen after the db connection is closed but before the request is returned. At this point, I could do a external connection and combine the resulting data to modify the response object, perhaps in an async thread like blobs uses. If I really wanted to write or read more data I could request a new thread and db connection at that point.


> 
>> I can see from the previous post, as there is no checkout semantics
>> in zodb,
> 
> I don't know what "checkout semantics" means.

As in the ZODB protocol doesn't have a call you have to make "before" you write to an object. You just write to the object and afterwards flag as changed (if needed). So there isn't a way to block at the point of writing. Malthe's database had an explicit checkout action so you weren't allowed to mutate anything until you checked it out presumably. Not something you can introduce into ZODB.

> 
>> you are free to write anytime so there is no sane way to block at the point
>> someone wants to write to an object, so it wouldn't work.
> 
> ZODB provides a very simple concurrency model by giving each
> connection (and in common practice, each thread) it's own view of the
> database. If you break that, then you're injecting concurrency issues
> into the app or in some pretty magical layer.
> 
>> You perhaps could have a single read only db connection which is
>> shared?
> 
> But even if the database data was only read, objects have other state
> that may be mutated.  You'd have to inspect every class to make sure
> it's thread safe. That's too scary for me.
> 
> Jim
> 
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
> Jerky is better than bacon! http://zo.pe/Kqm