• setsubyou@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    ·
    14 hours ago

    That’s funny because I grew up with math teachers constantly telling us that we shouldn’t trust them.

    Normal calculators that don’t have arbitrary precision have all the same problems you get when you use floating point types in a programming language. E.g. 0.1+0.2==0.3 evaluates to false in many languages. Or how adding very small numbers to very large numbers might result in the larger number as is.

    If you’ve only used CAS calculators or similar you might not have seen these too since those often do arbitrary precision arithmetics, but the vast majority of calculators is not like that. They might have more precision than a 32 bit float though.