4 Replies Latest reply on Dec 21, 2014 1:59 PM by philmodjunk

    Scientific notation dilemma

    calexmac

      Title

      Scientific notation dilemma

      Post

      After experimenting a bit, I don't think I understand how FMP implements scientific notation.

      I have a lot of mathematical calcs in a script that occasionally return very small numbers, e.g. 0.000000000123 which FMP kindly expresses as 1.23e-10.

      These are valid numbers with respect to the calculation and I want to to work with them.  But this is where I am looking for some help.

      Question 1: is a scientific notation value a text string or a number?

      The numeric calcs return their results in variables which are then put into a number field.  When FMP calculates a number that is very small it forces it into scientific notation, which then appears to behave as a text field.  For example, suppose I check if the number above is < or > than 0 using the following code:

      Case (
          $vInput < 0 ; "<0" ; 
          $vInput = 0 ; "=0" ;
          ">0"
      )

      ...then if $vInput is set to 0.000000000123 the code above returns ">0" which leads me to think that FMP is actually checking if 1.231 (i.e. FMPs scientific notation version of my decimal but with the "e-" removed) is < or > than 0.

       

      So Question 2 has to be: How do I execute comparisons with very small and large numbers that FMP wants to express in scientific notation?

      Obviously, a calculated result of 0.000000000123 is < 0!

       

      Final Question 3: How do mathematical operations work with such numbers?

      For example, suppose you set a variable $vInput to 0.0000000000002, which FMP will express as 2.0e-13 and you then display (using a show custom dialog) the result of $vInput * $vInput.  FMP will display "0", when the answer is surely 4.0e-26.

       

      Any thoughts much appreciated.

        • 1. Re: Scientific notation dilemma
          philmodjunk

          1) If this value is in a field the field type determines whether this is a number or text. If this is a calculation field, the result type setting specifies whether the value is stored as number or text.

          If you set a variable to this value, it's a bit less obvious as it is the type of calculation that returns the type used to store the value. At times, it is necessary to use GetAsNumber ( $Variable ) to be sure that FileMaker treats the value as a number.

          But in all cases, the format of the number does not determine the data type. A given value such as your scientific notation value will display in either the default setting you see now or in a data format that you choose for it. The data format option selected in the inspector does not change how the value is stored, only in how it is displayed.

          2) 1.233e -10, IS greater than zero as this is not a negative number. The minus sign specifies a negative exponent not a negative mantissa. -1.233e -10, on the other hand, would be less than zero. Note the minus sign at the beginning of this number used to make the mantissa negative.

          3) Filemaker stores all floating point values the same way not matter how they are displayed via different display formats and you would use them in calculations in exactly the same format.

          • 2. Re: Scientific notation dilemma
            calexmac

            Thanks for your answer.

            Re 1: All of the calculations are occurring in variables and this is perhaps where I am running aground. I would expect the data type to be independent of the representation.  I still don't understand the behaviour of GetAsNumber when used on variable with a tiny number in it... I have always presumed it means that If I set a variable say $vTest to GetAsNumber, the results of GetAsNumber ( $vTest ) will always be treated as a number, not the default text type that a variable value is expected to evaluate to.

            In that case, if I set a variable say $vTest to 0.0000000000002, then why is the calculation GetAsNumber ( $vTest ) * GetAsNumber ( $vTest ) displaying (via the show custom dialog) as 0 for me?

            Re 2: my mistake on the comparison! Duh! Too many solo hours trying to get to the bottom of a problem!

             

            thanks

             

             

            • 3. Re: Scientific notation dilemma
              Jade

              You are exceeding FMP's default precision with that calculation ( 2e-13 * 2e-13 = 4e-26).  Hence it rounds (or truncates) the result to 0.

              Use the SetPrecision function { e.g. SetPrecision( 2e-13 * 2e-13 ; 30 ) } to display the correct answer.

              • 4. Re: Scientific notation dilemma
                philmodjunk

                if I set a variable say $vTest to 0.0000000000002, then why is the calculation GetAsNumber ( $vTest ) * GetAsNumber ( $vTest ) displaying (via the show custom dialog) as 0 for me?

                Just as a calculator has a limited number of digits that it can work with and when a value exceeds that value, it rounds the value, so does computer software such as FileMaker. I would assume that a value of

                0.00000000000000000000000004

                Exceeds that limit and is rounded to zero. I have not researched that limit with FileMaker and if this value does not appear to exceed that limit, you may want to report this as a possible bug over in Report An Issue.