Page 1 of 1

resolution of a float?

Posted: Wed Oct 17, 2012 9:46 pm
by TedWalther
I have been converting some JavaScript code to newLISP. It is detailed astronomical calculations. Finally I notice something. The floats in newLISP are only 10 decimal digits long, while those in JavaScript are 16. I looked into the IEEE standard; there is 52 bits in which to store an integer without any loss of precision. That matches the output of JavaScript.

Lutz, are we using IEEE 64 bit floats in newLISP by default? I was hunting some bugs down for hours now, until I noticed this difference in precision.

Re: resolution of a float?

Posted: Wed Oct 17, 2012 10:15 pm
by Lutz
newLISP uses IEEE-754 double floats. The 10 digits seen are only the default display resolution. Use the 'format' function to display more digits. See also in the distribution the file qa-specific-tests/qa-float for some other IEEE floating point specifics.

Re: resolution of a float?

Posted: Wed Oct 17, 2012 10:37 pm
by TedWalther
That is a relief, thanks for explaining.