[ZODB-Dev] BTree bug on a 64 bit platform.

[John Barratt]

I got to wondering the otherday how IOBTrees cope with being moved 
between 32 and 64 bit platforms in the ZODB when the int's are different 
sizes.  I wanted to create a BTree on a 64bit platform using 64bit int 
keys...

This led me to some tests which show some inconsistency between the 
usage of BTree's on the two platforms.  This *could* be a 
system/compiler specific bug, but I only have one type of system to test 
on (AMD 64, Suse 9.0 64bit, Python 2.3.4).

Take the code :

from BTrees.IOBTree import IOBTree
t = IOBTree()
k = pow(2, 35) ## Create a > '32bit' int.
v = 'Value for key %i' % (k,)
t[k] = v
print "Input Key=%i\nType='%s'\nValue='%s'."% (k, type(k), v)
print
for k in t.keys():
     print "Output Key=%s\nType='%s'\nValue='%s'." % (k, type(k), t[k])

On a 32bit platform it rightly generates an exception due to the long 
being given as a key to the IOBTree :

Input Key=34359738368
Type='<type 'long'>'
Value='Value for key 34359738368'.
Traceback (most recent call last):
   File "showBTreeBug.py", line 7, in ?
     t[k] = v
TypeError: expected integer key

On the 64bit system it silently converts the int key to a '32 bit' key :

Input Key=34359738368
Type='<type 'int'>'
Value='Value for key 34359738368'.

Output Key=0
Type='<type 'int'>'
Value='Value for key 34359738368'.

I haven't looked in the core ZODB code to see if this is a potential 
problem (and it's probably a bit beyond me ATM), but it may be that 
since the keys are generated within the C code it will never be a 
problem there?

The only place in the python code I could immediately see the 
possibility of a problem is for incorrect indexing with the DateTime 
index for dates greater than around the year 4050.  These would be 
indexed as being around 'the start of time'.  Catalog rid's are 
generated from within a range that can always be represented in < 32bits 
so are OK.

Otherwise the main problem area would be in '3rd party' code (like 
mine!).  I also would imagine the same sort of problem would exist with 
other types of structures such as IIBTree etc.

Am I missing something here, or is this a bug that could one day really 
bite due to data loss from overwriting key values in BTrees?

Can this situation be dealt with 'nicely' other than changing the code 
to raise an exception if on a 64bit platform you try to use a > '32bit' 
int as a key?

This is the only way I can see to maintain portability of Data.fs files 
between 32 & 64 bit platforms.  Apologies if this has been covered 
before, but I could find no immediate reference.

Cheers,

JB.