Simple Math

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Joshua A. Booker - 19 Sep 2005 17:18 GMT

Hi there,

Why is int(1.4/.05) = 27?
It should be 28.

TIA,
Josh

Reply to this Message

Marshall Barton - 19 Sep 2005 19:08 GMT

>Why is int(1.4/.05) = 27?
>It should be 28.

Apparently(?), one (or both) of those number can not be
represented exactly as a binary floating point number. The
Int (and Fix) functions blindly discard any fractional part
regardless of how close the value is to an integer value.
You can avoid this by adding a small amount to the value
before the truncation:
Int(1.4/.05 + 1E-8)

If you want the value to be rounded when you convert it to
an integer, use CInt or CLng.

Signature

Marsh
MVP [MS Access]

Reply to this Message

Joshua A. Booker - 19 Sep 2005 20:41 GMT

Marsh,

I'm using the following code to round one number by another increment. I
can't get it to return the correct results when it's passed a 'whole'
(relative to the increment) number.

For example:
Rnd2Num(1.4,.05,vb_rounddown) = 1.35

In this case, because 1.4 is divisible by 0.05 I want it to return 1.4. The
line:
If CInt(Temp) = Temp Then
is supposed to catch this case.

Function Rnd2Num(Amt As Variant, RoundAmt As Variant, Direction As Integer)
As Double
On Error Resume Next

Dim Temp As Double
Temp = Amt / RoundAmt

If CInt(Temp) = Temp Then
Rnd2Num = Amt
Else
If Direction = vb_rounddown Then
Temp = Int(Temp)
Else
Temp = Int(Temp) + 1
End If
Rnd2Num = Temp * RoundAmt
End If
'Debug.Print Rnd2Num
End Function

I tried adding a small amount.
Any suggestions?

TIA,
Josh

> >Why is int(1.4/.05) = 27?
> >It should be 28.
[quoted text clipped - 9 lines]
> If you want the value to be rounded when you convert it to
> an integer, use CInt or CLng.

Reply to this Message

Marshall Barton - 20 Sep 2005 16:07 GMT

Your routine is too sensitive to the inaccuracies of the
floating point representation of many numbers.

I Googled the newgroup archives on this topic and found a
couple of gems among the heaps of gravel. Unfortunately, my
ISP shut down before I could capture the thread's URL.

Anyway, the algorithm that seemed to be the best was Lyle
Fairfield's simple one liner:

Format(Amt / RoundAmt, "0") * RoundAmt

I'll leave it to you to work out how to fit your direction
argument into it ;-)

Signature

Marsh
MVP [MS Access]

>I'm using the following code to round one number by another increment. I
>can't get it to return the correct results when it's passed a 'whole'
[quoted text clipped - 41 lines]
>> If you want the value to be rounded when you convert it to
>> an integer, use CInt or CLng.

Reply to this Message

Roger Carlson - 19 Sep 2005 19:09 GMT

Floating point math can be off by miniscule amounts. Probably it is
returning something like 27.99999999, which the Int function converts to 27.
The ususual solution is to use Currency datatypes, which are more accurate.
Exactly how are you using this?

Signature

--Roger Carlson
Access Database Samples: www.rogersaccesslibrary.com
Want answers to your Access questions in your Email?
Free subscription:
http://peach.ease.lsoft.com/scripts/wa.exe?SUBED1=ACCESS-L

> Hi there,
>
[quoted text clipped - 3 lines]
> TIA,
> Josh

Reply to this Message