resolution of a float?

Q&A's, tips, howto's
Locked
TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

resolution of a float?

Post by TedWalther »

I have been converting some JavaScript code to newLISP. It is detailed astronomical calculations. Finally I notice something. The floats in newLISP are only 10 decimal digits long, while those in JavaScript are 16. I looked into the IEEE standard; there is 52 bits in which to store an integer without any loss of precision. That matches the output of JavaScript.

Lutz, are we using IEEE 64 bit floats in newLISP by default? I was hunting some bugs down for hours now, until I noticed this difference in precision.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: resolution of a float?

Post by Lutz »

newLISP uses IEEE-754 double floats. The 10 digits seen are only the default display resolution. Use the 'format' function to display more digits. See also in the distribution the file qa-specific-tests/qa-float for some other IEEE floating point specifics.

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: resolution of a float?

Post by TedWalther »

That is a relief, thanks for explaining.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Locked