RE-POST: Incorrect formula result/ decimal places

[QuantumLeap]

This message is a re-post from 4-28-06.

I have a query in which I calculate the difference between two measurements,
Left and Right.

My formula reads Diff:

-

This should be simple enough, but I am getting strange results. For example:
0.3 - 0.2 = 0.100000008940697

My operators who enter the measurements have maximum of three digits past
the decimal place, so the math is incorrect. I do not understand why this
is happening.

I would like the result of the formula to have the correct answer and three
digits to the right of the decimal and have formatted the column for this
(Properties>General>Decimal Places>3), but am getting the same results. I
even tried formatting the Left and Right columns the same way.

Does anyone have any suggestions on how I can correct this problem?​

 

[KARL DEWEY]

Try setting your format to Single. Single is only 7 decimal places and will
romove the trailing zero.

 

[QuantumLeap]

My format is already single, unless I am setting it incorrectly.

In the table's design view: Select row>Field Size>Single.

Try setting your format to Single. Single is only 7 decimal places and will
romove the trailing zero.

This message is a re-post from 4-28-06.

I have a query in which I calculate the difference between two measurements,
Left and Right.

My formula reads Diff:

-

This should be simple enough, but I am getting strange results. For example:
0.3 - 0.2 = 0.100000008940697

My operators who enter the measurements have maximum of three digits past
the decimal place, so the math is incorrect. I do not understand why this
is happening.

I would like the result of the formula to have the correct answer and three
digits to the right of the decimal and have formatted the column for this
(Properties>General>Decimal Places>3), but am getting the same results. I
even tried formatting the Left and Right columns the same way.

Does anyone have any suggestions on how I can correct this problem?
​

​

​

 

[KARL DEWEY]

Format your display - query, form or report.

My format is already single, unless I am setting it incorrectly.

In the table's design view: Select row>Field Size>Single.

Try setting your format to Single. Single is only 7 decimal places and will
romove the trailing zero.

This message is a re-post from 4-28-06.

I have a query in which I calculate the difference between two measurements,
Left and Right.

My formula reads Diff:

-

This should be simple enough, but I am getting strange results. For example:
0.3 - 0.2 = 0.100000008940697

My operators who enter the measurements have maximum of three digits past
the decimal place, so the math is incorrect. I do not understand why this
is happening.

I would like the result of the formula to have the correct answer and three
digits to the right of the decimal and have formatted the column for this
(Properties>General>Decimal Places>3), but am getting the same results. I
even tried formatting the Left and Right columns the same way.

Does anyone have any suggestions on how I can correct this problem?
​

​

​

​

 

[John Spencer]

Floating point numbers often do things like this in calculations. Just as
in the decimal system we cannot represent 1/3 totally accurately .33333
forever, there are numbers that cannot be accurately stored in binary
systems so we get approximations. In your case, changing the field type to
currency will work. Currency is a fixed number type with 4 decimal places -
so it is accurate to 4 decimal places.

Your other choices are to round the result or to set a display format to
show a maximum of three decimal places. Also, you might try use Double as
the field type instead of single. That often works due to the increased
precision of the double type (more bytes to represent the decimal portion -
but it is still a floating point number and can exhibit the same problem
(just not as often)).