[ZODB-Dev] Transaction note interface

[Tim Peters]

[Tim Peters]
>> It's clear from the implementation that notes() doesn't expect to see
>> more than a few relatively short strings per transaction (the
>> implementation requires time quadratic in the number of notes added in a
>> transaction).

[Christian Theune]
> Is it number of notes or size of the accumulated note?

Some of each, really.

> Number of notes would be quadratic in the notes() function, but I can't
> see it there ...

Right here:

    self.description = "%s\n\n%s" % (self.description, strip(text))

That copies all of .description so far, each time note() is called.  If
note() is called n times, and the strip(text) thingies were saved in
text[0], text[1], ..., text[n-1], then total runtime is proportional to

    sum(len(text[i])*(n-i) for i in range(n))

That is, text[0] is copied n times, text[1] n-1 times, etc.

Routines that *expect* to concatenate lots of strings append the pieces to a
list instead, and do a join() once at the end.

>> Maybe it would be better if storages with a limit on description size
>> automatically cut back (presumably rare) over-long descriptions instead
>> of raising an exception long after it's too late for the user to "do
>> something" about that.  Ditto for the user metadata.

> Well. I think this would be a feasable option, as the moment this caught
> me was during the automated generation of content for a plone site that
> involved bout 150 (content) objects + workflow actions. The one thing I
> was surprised was that the note() function is accumulating.

I don't know why it does that.  At least it's documented in ZODB 3.3 (in an
ITransaction interface, docs reverse-engineered from staring at the
implementation).  It's hard to know whether it matters.

>> There's also Transaction.setExtendedInfo().  I don't know why it exists,
>> and it appears Zope never uses it.  It creates a dict under the covers,
>> and that also gripes if the pickle of the dict is >= 64KB.  If anything
>> does use this, auto-reducing its size probably isn't justifiable.  I'm
>> not sure anyone would care if FileStorage cut arbitrary stuff out of a
>> giant description or user path.

> I _think_ I saw setExtendedInfo beeing used in a couple of 3rd party
> products.

Ya, and I later found out Zope3 actually uses it.

> The thing that might really be annoying to people is that Storages are
> the problem here. It sounds like a rather arbitrary limit. (The unit
> tests should have catched that problem but DemoStorage doesn't have that
> limit :/)

It's certainly arbitrary!  FileStorage simply has a 2-byte field to record
how long the description is (and another to record the length of user, and
another to record the size of the extension pickle).  Those could be
boosted, but that's an awful lot of work for such a little thing (old code
literally couldn't read new FileStorages anymore, and current code would
have to grow complication to read both old and new formats).

> I'm currently not sure what would be a good way:
>
> - keeping everything as is and educate people about that limit in some
> place where note() is documented -

I would do that, but I think Jim would object, since it's not a limitation
of the Transaction interface (to which note() belongs) but has solely to do
with storage implementations.  But if it's not documented with note(),
nobody will find out about it.

> have it implicitly reduce the note, but fail on large extended info -
> cut back both, extended info and the note
>
> I'm not feeling too good about cutting back a large extended info.

I think that's right out.  It's a pickle, it has structure, there's no way
to guess which parts are and aren't "important".  Nobody has a 64K .user, so
that's a red herring.  Given the only uses for .description I've ever seen,
it wouldn't be a real loss to chop stuff "out of the middle" -- the first
and last few note()s are the interesting ones, and the problem (when it
occurs) is due to the sheer number of notes.

> Well.  Even on cutting back the note, as software or people might rely
> on them to actually store what you give them or complain.

I don't think so.  AFAICT, these are "write only" gimmicks:  there is no API
that allows you to *read* this info again.  Indeed, you started this thread
with the observation that there's no blessed way to retrieve the description
even from the currently active transaction.  After the transaction is in the
database, there's nothing in the documented storage API that allows
retrieving this stuff again.  There are internal tools (like fsdump) that
retrieve and display the user and description fields via exploiting
undocumented internals, but AFAIK there's nothing anywhere that displays
"extended info".  So the transaction meta-info appears to be all in the
nature of forensic clues in case a database dies or goes insane.