[ZODB-Dev] blobs missing with relstorage and small blob cache dir

[Shane Hathaway]

On 02/28/2011 08:19 AM, Maurits van Rees wrote:
> I wonder if there is some code that mistakenly throws away the wrong
> blob; it should throw away the oldest one I'd think, and not the one
> that it just loaded.
>
> The workaround is probably just to not set the blob-cache-size this low.
>    But I tried this low setting to work around a different problem, which
> I will report in a different post.

This reveals how naive the blob cache management is in both RelStorage 
and ZEO.  It is smart enough to use the atime of files (thus you should 
not disable atime on a filesystem that hosts a blob cache), but the 
cache management does not guarantee downloaded files are kept long 
enough to actually read them.

The usual Unixy solution is to pass around an open file descriptor 
instead of a filename, so that even if the blob file is deleted, the 
thread that asked for a file is sure to be able to read it.  However, 
the ZODB blob APIs demand a filename, not an open file, so that solution 
is not feasible.

We could make this kind of exception retryable, or we could make the 
blob cache management refuse to delete files within some time limit, but 
those solutions would be workarounds too.  In the end I think it's best 
to just tell everyone that the minimum reasonable blob cache size is 
some multiple of your largest blob.  I wouldn't set it lower than 1 GB. 
  I would set it much larger for hosting videos or CD/DVD images.

Shane