Calculation accuracy

S

Sathish

When i tried calculating a simple math, i found a rounding issue in excel:

I made the following computation by having each value in a cell and inputed
formula to compute the answer:

A1 = 19978.39
B1 = 21456.06
C1 = 1477.67
D1 = A1 - B1 + C1 = -0.00000000001819 (instead of ZERO).

I dont understand the reason for this type of rounding eventhough the
amounts involved are relatively round.

May i have the esteemed views of this forum??

Thanks
 
B

Bernard Liengme

This has to do with the way computer (not just Excel) store decimal numbers
(numbers to base 10) in binary format (numbers to base 2). For details read:
INFO: Visual Basic and Arithmetic Precision
http://support.microsoft.com/defaul...port/kb/articles/Q279/7/55.ASP&NoWebContent=1

(Complete) Tutorial to Understand IEEE Floating-Point Errors
http://support.microsoft.com/defaul...pport/kb/articles/Q42/9/80.ASP&NoWebContent=1

What Every Computer Scientist Should Know About Floating Point
http://docs.sun.com/source/806-3568/ncg_goldberg.html
Go to source>>
http://www.cpearson.com/excel/rounding.htm

best wishes
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Top