5 Replies Latest reply on Dec 28, 2011 3:04 PM by user15307

    On Mac OX Lion, SetPrecision() returns only 8 digits of precision after the decimal

    MarioMarceau

      Summary

      On Mac OX Lion, SetPrecision() returns only 8 digits of precision after the decimal

      Product

      FileMaker Pro

      Version

      FileMaker Pro Advanced 11.0v4

      Operating system version

      Mac OS X Lion 10.7.3 (11D24)

      Description of the issue

      When I use the SetPrecision in a calculation field, here's what happens:

      Ex: SetPrecision ( Pi ; 20 )

      1) If I set the calculation result as "Number", I get only 8 digits of precision after the decimal point;

      2) If I set the calculation result as "Text", I get as many digits of precision (up to 400) without any problem;

      This occurs in the Table View as well as in the Form View (even if I stretch the fields long enough to accommodate the number of digits I want.

      I USED THREE DIFFERENT DATA FORMATTING WHILE IN EDIT LAYOUT MODE: "GENERAL", "AS ENTERED", AND "DECIMAL". ALL GAVE THE SAME OUTCOME.

      Steps to reproduce the problem

      1) Use a machine with OS X Lion 10.7.3;

      2) Set a calculation field in a file and call it say, Pi_Precision

      3) In the calculation field, enter the following formula: SetPrecision ( Pi ; 20 )

      4) In the bottom of the calculation field window, set the Calculation result to "Number";

      5) Go back in table mode and stretch your field to accommodate the precision of 20;

      6) The result should show 3.14159265358979323846 but it truncates it to 8 digits of precision, UNLESS you click into the field, at which point it will show you the entire string;

      7) Go back to change the Calculation result to "Text";

      8) The result shows the entire result correctly, without having to click into the field, but at this point, the field is recording a text string as opposed to a number.

      Expected result

      No truncation when the result is set to return a Number.

      Actual result

      It truncates the number to 8 digits of precision when setting the result to return a Number.

      Exact text of any error message(s) that appear

      None

      Configuration information

      Someone on the TechNet forum told me that this issue didn't occur in OS X Snow Leopard. I have no point of reference since I don't have a copy of Snow Leopard.

      I suggest that it could be induced with the changes made by Apple in OS X Lion which is quite a departure from Snow Leopard.

      Workaround

      The only way to see the whole number (When the field is set to return a Number is to click into the field. I believe this may inconvenience scientists and engineers who may rely on a large degree of precision.)

      IMPORTANT:

      I RECREATED A NEW FILE BASED ON A MEMBER'S COMMENT: I WAS ABLE TO GET IT TO WORK PROPERLY THIS TIME.

      THE ONLY DIFFERENCE I CAN RECALL IS THAT WHEN I CREATED THE FIELD FOR THE CALCULATION IN MY FIRST TRY, I STRETCHED THE FIELD TO ACCOMMODATE THE NUMBER OF DIGITS ONLY AFTER I DECLARED THE CALCULATION.

      IN THE NEW FILE, I DID THAT STEP BEFORE DECLARING THE CALCULATION. IT WORK PROPERLY THIS TIME.

      IT MAKES ME WONDER IF THERE'S A GLITCH THAT TRUNCATES THE RESULTS IF WE FAIL TO STRETCH THE FIELDS TO ACCOMMODATE THE CALCULATION PRIOR TO DECLARING IT.

      PLEASE BE ADVISED THAT THERE WERE OTHER FIELDS IN THE FIRST (FAULTY) FILE. FIELDS SUCH AS A NUMBER FIELD, THEN A FEW CALCULATION FIELDS FOR FLOOR , CEILING, ROUND OF THAT FIRST NUMBER FIELD. I ADDED THE Pi FIELD WITH IT'S OWN CALCULATION AT THE END OF THE TABLE.