[ZODB-Dev] OID length

[Tim Peters]

[Florent Guillaume]
> I know this has been raised several times in the past, but I'm not sure
> of all the details...

You should believe whatever Jim tells you ;-)

> What layer of ZODB, if any, has a restriction that OIDs be 8-byte
> strings. Is it the persistence module? Or just FileStorage?

FileStorage uses oids that are exactly 8-byte strings.  The implementation
of the FileStorage index (built on the special-purpose fsBTree type) relies
on that heavily.

The UML model for ZODB 3 claims that "In ZODB 3, object identifiers are
8-byte strings" generally, but at least Shane Hathaway's APE got away with
using (at least for some time) much longer strings.  I'm not sure where the
code actually relies on "stringness", though -- looks like most of it would
work provided oids were totally ordered, hashable, immutable, and
pickle-able (so that oids can be used as keys in Python dicts and in
BTrees).

Some "auxiliary" code would definitely break if they weren't strings.  For
example, ZEO cache tracing has its own trace-file format, which originally
blew up left and right when Shane tried to use it with APE.  That got
generalized (and, alas, the trace files got bigger) to live with APE's (at
the time) variable-length string oids, but was not generalized to live with
non-string oids.

> Also is there a reason (performance, space?) for this restriction,

Certainly:  any storage is in the business of managing a potentially huge
number of oids, and unused generality is rarely free.  The FileStorage index
buys a lot of RAM efficiency from sticking to 8-byte strings.  I _think_ APE
eventually moved back to using "small" strings too.

> and would anyone be interested in seeing it removed.
>
> I'm asking because I'm planning a pretty complex storage layer, which
> would benefit from more complex OIDs (for instance bigger strings, or
> tuples of (str, int)).

An oid is a low-level implementation thingie, and IMO that's probably not
the right place to put complexity.  If you don't expect to have more than
2**N objects, an oid that requires more than N bits is in some sense
extravagant ;-).  In any case, you should be able to use "bigger strings"
now without changing anything.