Another numeric formatting anomaly

J

JoeU2004

myMod(12346789-0.0000004,1) returns a binary representation which is exactly
0.999999597668647,76611328125. (The comma demarcates the first 15
significant digits.)

But when the cell (A1) is formatted as Number with 15 decimal places, Excel
displays 0.999999597668647 (!).

In contrast, myCstr(A1) returns 0.999999597668648 as expected.

These examples are for Excel 2003 11.5612.5606, part of MS Office Sm Busn Ed
2003 on MS Win XP SP3, and VB 6.3 version 9969, VBA Retail 6.4.8896, Forms3
11.0.5601.

The UDFs are:

Function mymod(a As Double, b As Double) As Double
mymod = a - b * Int(a / b)
End Function

Function myCstr(arg As Double) As String
myCstr = CStr(arg)
End Function
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top