Skip to content

Incorrect results from BigDecimal.divide depending on prevision and rounding mode. #318

Open
@gsteckman

Description

@gsteckman

Describe the bug
When using BigDecimal.divide with a DecimalMode, the results are sometimes incorrect.

To Reproduce
Run this code:

@Test
    fun testDivision() {
        val a = 1.08589200008919E+4
        val b = 1.085892300018812E+5

        val aOverB = a / b
        println("a/b=${aOverB}")

        val a2 = BigDecimal.fromDouble(a)
        val b2 = BigDecimal.fromDouble(b)

        RoundingMode.entries.filter { it != RoundingMode.NONE }.forEach { mode ->
            println("mode: $mode")
            listOf(5L, 6L, 7L).forEach { precision ->
                val a2OverB2 = a2.divide(b2, DecimalMode(precision, mode))
                println("\tprec: $precision, a2/b2=${a2OverB2}")
            }
            println()
        }
    }

Expected behavior

I expect all results to be similar to the Kotlin Double result, subject to the precision and last digit rounding behavior. Sometimes the result is approximately 10x different from the expected result.

The result from the above code is:

a/b=0.09999997237943192
mode: FLOOR
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=9.99999E-2
prec: 7, a2/b2=9.999997E-2

mode: CEILING
prec: 5, a2/b2=1.0E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999998E-2

mode: AWAY_FROM_ZERO
prec: 5, a2/b2=1.0E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999998E-2

mode: TOWARDS_ZERO
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=9.99999E-2
prec: 7, a2/b2=9.999997E-2

mode: ROUND_HALF_AWAY_FROM_ZERO
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999997E-2

mode: ROUND_HALF_TOWARDS_ZERO
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999997E-2

mode: ROUND_HALF_CEILING
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999997E-2

mode: ROUND_HALF_FLOOR
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999997E-2

mode: ROUND_HALF_TO_EVEN
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999997E-2

mode: ROUND_HALF_TO_ODD
prec: 5, a2/b2=9.9999E-2
prec: 6, a2/b2=1.0E-2
prec: 7, a2/b2=9.999997E-2

Platform

Seen on JVM 17.0.8. I didn't test other platforms but it may exist on other platforms as well.

Using com.ionspin.kotlin:bignum:0.3.10.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions