Can some more experienced guys explain this strange error I found
today?
I was getting strange amounts when I was loading table data with my C#
script.
As it turns out the problem is a
different output from similar
functions:
string
amount_in_string = "1234567,15";
double amount_in_double =
double.Parse(amount_in_string);
float amount_in_float =
float.Parse(amount_in_string);
//amount_in_double =
1234567.15;
//amount_in_float =
1234567.13;
Why do I
get such a different result when float and double are similar types (floating point).
Can the precision make a difference with small amounts like
these?
When
“1234567.15” is converted to double
, the result is the closest
value representable in double
, which is
1234567.1499999999068677425384521484375. Although you report in the question that the
value is 1234567.15, the actual value is 1234567.1499999999068677425384521484375.
“1234567.15” would be displayed when the value is displayed with a limited number of
decimal digits.
When “1234567.15” is converted
to float
, the result is the closet value representable in
float
, which is 1234567.125. Although you report the value is
1234567.13, the actual value is 1234567.125. “1234567.13” may be displayed when the
value is displayed with a limited number of decimal
digits.
/>
Observe that 1234567 exceeds 1,048,576, which is
220. The 32-bit floating-point format used for
float
uses 24 bits for the significand (fraction portion of the
number). If the high bit of the significand represents 220, the
low bit represents 220−23 = 2−3 = ⅛. This
is why you see “1234567.15” converted to a value rounded to the nearest
eighth.
No comments:
Post a Comment