[ZODB-Dev] Advice on ZODB with large datasets

[Shane Hathaway]

AFoglia at princeton.com wrote:
> We have a large dataset of 650,000+ records that I'd like to examine 
> easily in Python.  I have figured out how to put this into a ZODB file 
> that totals 4 GB in size.  But I'm new to ZODB and very large databases, 
> and have a few questions.
> 
> 1. The data is in a IOBTree so I can access each item once I know the 
> key, but to get the list of keys I tried:
> 
> scores = root['scores']
> ids = [id for id in scores.iterkeys()]
> 
> This seems to require the entire tree to be loaded into memory which 
> takes more RAM than I have.

What kind of objects are the values?  They should probably extend 
Persistent.  Just my $0.02.

Shane