[ZODB-Dev] Errors running a Stress Test

[Brandon Ehle]

Tim Peters wrote:
> More on Brandon's stress test:  a few of these appear to be expected
> consequences of using pack in an unnatural way, starting with all the
> repetitions of this kind of traceback:
> 
> The test has many threads and clients all slamming on a single OOBucket (oid
> 1, which always occurs in these tracebacks), and every few minutes each
> client does a pack to current time.  What happens:
> 
>     transaction T1 begins; the OOBucket is at revision R1
>     transaction T2 begins; the OOBucket is still at revision R1
>     transaction T2 commits; the current revision becomes R2
>     a pack to current time occurs, and removes revison R1 (it's
>         "ancient" history by now)
>     transaction T1 tries to commit revision R3
>     conflict resolution tries to read revision R1 (that's oldSerial
>         in the traceback above), but it no longer exists
>     POSKeyError results
> 
> This is an unexpected event in conflict resolution because it shouldn't
> normally happen when pack is used -- pack() is usually used once a week or
> month, and you usually pack to a day or so in the past.  It's extremely
> unlikely then that a transaction currently in progress began before the pack
> time, so it's also extremely unlikely then that any transaction in progress
> will need to access a non-current revision before the pack time (non-current
> revisions before the pack time are the ones destroyed by a pack).
> 
> The kind of POSKeyError traceback above can't be prevented if a pack does
> destroy non-current revisions eventually needed by transactions currently in
> progress.  In real life, these don't occur, but when they do <wink> they're
> not harmful -- it gets reported back to the client as a ConflictError, and
> the client retries the transaction then.  The next time it will start with
> the then-current revision pack didn't destroy.
> 
> That's also an artifact of the test driver, and won't change.  Because pack
> in real life is invoked rarely, and can take a very long time (hours, even a
> day for a huge .fs), it's unreasonable for the implementation to "stack up"
> pack requests and just sit there waiting for them to become runnable.  It's
> also unreasonable to ignore a pack request -- if a pack ain't gonna happen,
> an exception is appropriate.  Of course the test driver could avoid this by
> invoking pack from only one client at a time (or from only one client
> period).
> 

I guess that pack means something different to me, because on average, 
the FileStorage is growing on me by about 50MB / hour on a machine with 
a fast but small 10,000 RPM drive.  Thus my stress test is designed to 
mimic my regular usage, but uncover errors faster.

I currently need to pack several times a day (about once a hour 
currently) to prevent two things:

1) Not run out of disk space.

2) To make the pack do less work in a single operation so that my 
transaction throughput doesn't suffer and cause conflicts.