AnsweredAssumed Answered

Scientific notation dilemma

Question asked by calexmac on Dec 21, 2014
Latest reply on Dec 21, 2014 by philmodjunk

Title

Scientific notation dilemma

Post

After experimenting a bit, I don't think I understand how FMP implements scientific notation.

I have a lot of mathematical calcs in a script that occasionally return very small numbers, e.g. 0.000000000123 which FMP kindly expresses as 1.23e-10.

These are valid numbers with respect to the calculation and I want to to work with them.  But this is where I am looking for some help.

Question 1: is a scientific notation value a text string or a number?

The numeric calcs return their results in variables which are then put into a number field.  When FMP calculates a number that is very small it forces it into scientific notation, which then appears to behave as a text field.  For example, suppose I check if the number above is < or > than 0 using the following code:

Case (
    $vInput < 0 ; "<0" ; 
    $vInput = 0 ; "=0" ;
    ">0"
)

...then if $vInput is set to 0.000000000123 the code above returns ">0" which leads me to think that FMP is actually checking if 1.231 (i.e. FMPs scientific notation version of my decimal but with the "e-" removed) is < or > than 0.

 

So Question 2 has to be: How do I execute comparisons with very small and large numbers that FMP wants to express in scientific notation?

Obviously, a calculated result of 0.000000000123 is < 0!

 

Final Question 3: How do mathematical operations work with such numbers?

For example, suppose you set a variable $vInput to 0.0000000000002, which FMP will express as 2.0e-13 and you then display (using a show custom dialog) the result of $vInput * $vInput.  FMP will display "0", when the answer is surely 4.0e-26.

 

Any thoughts much appreciated.

Outcomes