[ZODB-Dev] 0 is not really equal to my object

[Tim Peters]

[Derrick Hudson]
> Oh, I thought C used IEEE 754

The C89 standard says almost nothing about floating-point semantics.  C99
says more, but doesn't require 754-like behavior unless the vendor chooses
to implement an optional part of C99.

> and that 754 specified the bit pattern to represent infinity, etc.

Yes, but 754 is not a language standard.  It's a standard for a system of
binary fp arithmetic; it says nothing about how any of its features can be
gotten at from any programming language.  C89 in turn says nothing about 754
(although an optional part of C99 does).

> I thought at the byte-level the platforms were consistent,

All platforms I'm aware of with 754 HW have byte-compatible float and double
representations, modulo endianness differences, and disregarding some
conventions relating to signaling and quiet NaNs, and disregarding that some
HW doesn't actually support subnormals.  But again that's got nothing to do
with programming language semantics.

> just not in their conversion to and from strings.
>
> So why doesn't the ZODB allow storing platform-dependent representations
> (of course with some suitable warning to users)?

For a start, because nobody asked for it before.  Now that someone has
<wink>, my bet is that nobody will care enough to implement it -- and
there's no other case where ZODB tries to expose platform-dependent
behavior.

Anyway, it's not ZODB, it's cPickle that doesn't know how to deal with this
stuff, and pickles are platform-*in*dependent.  If you're willing to use
platform-dependent stuff, you can build non-portable strings using
struct.pack in "native" mode.  That's not an appropriate choice for pickles,
though (simply _because_ it's not portable -- and the portability of
FileStorage databases across platforms is a major feature -- although broken
in the presence of 754 endcases -- Python has no deliberate support for
those).

> >>> import pickle
> >>> pickle.dumps(float('inf'))
> 'Finf\n.'
>>>> import cPickle
> >>> cPickle.dumps(float('inf'))
> 'Finf\n.'
> >>> cPickle.loads(cPickle.dumps(float('inf')))
> inf
>
> This is actually what I expected.

Whether any of that appears to work is a platform-dependent accident (as is
all behavior in the presences of infinities, NaNs, or signed zeroes).  That
pickle cannot, for example, be unpickled on Windows.  In my opinion, it's a
bug that the picklers don't raise an exception here, because the pickle is
in fact not portable, and pickle does not intend to produce non-portable
stuff.  Pickle should raise an exception, or use its own
platform-independent representation of infinity.  But because C89 doesn't
say anything about infinities, there is no standard C function that can be
called to _detect_ that you passed an infinity.  Every C vendor on a 754 box
supplies their own, vendor-specific gimmick for this, and the current
pickling implementations simply ignore the issues here, letting the platform
C do whatever it does.

Best advice is "don't go there".  If you have to, you're on your own:  there
really is no non-accidental support in Python (let alone ZODB) for 754's
special values.

> ...
> I'm not surprised 'inf' isn't understood by msvcrt, but I am surprised
> they don't provide their own representation.  How did you get an infinity
> in to sprintf()?

double x = 1e300, then pass x*x to sprintf.  Whether that blows up depends
on whether the 754 overflow trap is enabled; it's disabled by default under
MS's C (and under almost all others as well), so sprintf() gets to do its
job.

...
>> ZODB adds a few tricks of its own, but in general can't work with any
>> object that cPickle can't work with on its own.

> ZODB could do a better job of reporting the problem.

Patches accepted.  Please realize that we don't get a dozen messages about
this per month here <wink -- but, really, yours is the first message of this
kind I've seen>.  Most people are familiar with pickle from Python first,
and so wouldn't even think about trying to use a name-rebinding singleton
idiom with a pickle-based approach.

You can google on "singleton pattern Python" for lots of other singleton
approaches, but I'm not sure any can help you across pickling (I would be
surprised if they could).

> I realized what was happening (duh!) when I studied how the ZODB loads
> the stored object:
...
> I suppose that 'print' should be replaced with raising a more appropriate
> exception.  I guess I only managed to find that issue because I made that
> sentinel (with the same name as the class) a string and string objects
> don't have a __basicnew__ attribute.

Frankly, I can't think of a realistic way for code around a failing
__basicnew__() call to produce an exception that's likely to be intelligible
to the end user.  It's too deep in the internals to guess that "hmm, maybe
they created a class, and then created a singleton instance of that class
with the same name as that class?" might be a sane thing to investigate.

More, it's generally true that pickling and unpickling exceptions are hard
to make sense of for end users (I'm often an end user myself, and have spent
many hours puzzling over exactly what a failing pickle dump or load is
complaining about).  So, my sympathy, but no offer of genuine relief.

...

>> Did you try defining a *non*-Persistent class to model the behavior you
>> want (an utterly ordinary Python class, preferably "new style")?

> So now you tell me that that is -supposed- to work!  :-)
>
> That was the first thing I tried.  (see the end of
> http://mail.zope.org/pipermail/zodb-dev/2005-February/008483.html)

Drat!  Sorry, that's still in my inbox -- if I can't make time to respond to
something within a day, chances are I won't get to it for a loooong time
after.  I remember looking at it long enough to see:

    PicklingError: Can't pickle <class
    'Products.ROSInfoSys.fields.infinite.Infinite'>: it's not the same
    object as Products.ROSInfoSys.fields.infinite.Infinite

at the end and spacing out with "egads, another incomprehensible
PicklingError -- I'll have to wait until I can make time to dig into that".
Although now that we know what it's saying, it's quite obvious <wink>.

> Hmm, now I understand the error I got then.  I was using that singleton
> pattern.  Now I'm back full-circle to being able to do what I wanted in
> the first place.  At least the result is good, and I learned quite a bit
> along the way :-).

Yup, it's been a fascinating journey <smile>.

> As far as value vs. reference .. value is good enough (hmm, I better make
> __eq__ test value not identity)

If you want a pair of these to compare equal, yes.

> although having more than one instance of this class isn't actually
> beneficial.

The tradeoffs are yours to make, but you already know that there are
barriers forcing _some_ tradeoffs here.