[ZODB-Dev] Sharing (persisted) strings between threads

[Malthe Borch]

On 8 December 2010 13:28, Jim Fulton <jim at zope.com> wrote:
>> With 20 active threads, each having rendered the Plone 4 front page,
>> this approach reduced the memory usage with 70 MB.
>
> Out of a total of what?

In my case out of 430 MB non-shared for the process.

> Note that if a process is CPU bound (as most dynamic Python apps
> should be), then there is little or no benefit in having multiple
> threads, due to the (damn) GIL.

The case I'm thinking of is when one thread is being used in a write
transaction, while another is doing a read.

If the database is bigger than the allowed memory usage, then I guess
threads can also ensure that requests for in-memory objects can be
served while some threads are blocked due to swapping and/or reading
pickles from disk.

> Except that you can't create wekrefs to strings or unicode.

I see. Maybe another scheme could be devised.

> Also, while interning is fine for an experiment, it's wasteful for
> strings that are rarely needed.

How so? As far as I can see, interning is still subjected to reference
counting. The only real difference is that a hash table is maintained
(fairly minimal memory use + probable computation of string hash).

> Sharing immutable data between threads is very appealing
> intellectually. I've certainly thoughtr about it a lot. In practice,
> I doubt the benefit will be worth the extra overhead (let alond the
> effort :).

I think if the case can be made for threading, then it's worth
pursuing. Alternatively, applications might put all non-trivial
strings into blobs, but I don't know if there's a non-trivial overhead
with that approach.

\malthe