[ZODB-Dev] Errors running a Stress Test

[Tim Peters]

[Tim sez pack is generally called infrequently, and that it's good to
 pack to some time in the (recent) past instead of to "right now";
 calling it frequently, and from multiple processes simultaneously, and
 packing to "right now", cause several of the odd tracebacks in Bradon's
 tests; and we don't have any plans to change those
]

[Brandon Ehle]
> I guess that pack means something different to me, because on average,
> the FileStorage is growing on me by about 50MB / hour on a machine
> with a fast but small 10,000 RPM drive.  Thus my stress test is
> designed to mimic my regular usage, but uncover errors faster.

FileStorage obviously wasn't designed for small disks, but I don't think
there's anything "wrong" with your usage.  You can still pack to some time
in the recent past (say, pack to 5 minutes ago, if transactions typically
take less than a minute), and you can still arrange to run no more than one
pack at a time.  Then you won't see the kinds of tracebacks explained in the
last mail.

> I currently need to pack several times a day (about once a hour
> currently) to prevent two things:
>
> 1) Not run out of disk space.

Packing to several minutes in the past should make little difference to
that.

> 2) To make the pack do less work in a single operation so that my
> transaction throughput doesn't suffer and cause conflicts.

I don't follow that one.  Pack time is roughly proportional to the sums of
the sizes of the "before" and "after" files.  If I pack 12 times in 12
hours, or one time in 12 hours, I get rid of the same total number of
expired objects, but pay repeatedly to copy the then-still-live objects in
the former case.  So I expect that *total* system burden would lessen by
packing less often.  OTOH, a large pack sucks up more disk cache, etc.  If
you've tried it both ways and measured it objectively, I'll agree to believe
whatever your clock said <wink>.