[ZODB-Dev] Feature Request 2381 : Persistent object iterator in Storage

[Greg Ward]

[Neil, in a private e-mail, breaking the news gently about my
 zodb_census script]
> Actually, it doesn't work.  It terminates too early.

The termination detection in zodb_census is pretty simple:

  oid = 0L
  expected_count = get_database().objectCount()
  while 1:
      # ... stringify oid, fetch the object, and process the object ...

      oid += 1
      if oid >= expected_count:
          sys.stdout.write("\rOID: %016x\n" % oid)
          print "census completed"
          break

get_database() simply returns our sole global instance of ZODB.DB.
objectCount() is a method of ZODB.DB, so I didn't have to code anything
there.

[Neil then refers to a database-walking script he wrote]
> I think commit_setstate.py works.  Unfortunately it only works for
> FileStorage.py.

Here's Neil's algorithm:

    try:
        oids = storage._index.keys()
    except AttributeError:
        raise SystemExit, "No _index attribute on storage, use FileStorage"
    oids.sort() # try to improve locality of reference
    # ...
    for oid in oids:
        # fetch and process the object

Can anyone vouch for either of these database-iteration algorithms?  Or
should we all start using the module Barry posted?  Is that module going
to be added to an official ZODB distribution one of these days?  Andrew,
should it maybe be added to your unofficial distribution?

        Greg