11 Replies Latest reply on Apr 18, 2017 8:35 AM by BobbyO'steen

    Calibri + unrecognizable character breaks FileMaker PDF Engine

    emory_brown

      Product and version: FileMaker Pro 15.0.3

      OS and version(s): macOS Sierra, Windows 10, Windows 7.

      Hardware: tested on Mac Mini, virtual machines, and windows PC.

      Description:

      When an unrecognizable character is included in a field, Save Records As > PDF will render anything in the Calibri font family completely screwed up--visually all letters from all data are stacked on top of each other, resulting in a blob.

       

      The first problem is FileMaker is allowing non-recognizable characters to be inserted into the database. The second issue is that the FileMaker PDF engine cannot handle these with certain fonts (Calibri is the only confirmed, there may be others.)

       

      How to replicate:

      Save records as PDF from example file. If you backspace out the last character in the first field, the PDF will then render normally (note: the cursor will not move, since there is an invisible character beyond the "SA").

       

      Workaround:

      Font Issue: Don't use Calibiri. However, the entire system is in Calibri because Arial broke PDFs in the last patch (15.0.2), and the workaround was to not use Arial in the mean time. This is sort of a vicious, very costly cycle at this point.

       

      Data issue: Find and replace all data in every field in every table with a custom function which utilizes Filter() function to ensure only letters, numbers, and symbols are allowed into the system. Additionally use a third-party plugin to capture clipboard data before it is pasted into the system and do the same. I've found that this workaround presents challenges when dealing with multiple languages, as all characters and symbols from every language's keyboard will need to be included. There is also an concern here with large data stores because sifting years of data entry including unrecognizable characters presents huge risk for introducing error.

        • 1. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
          TSGal

          emory_brown:

           

          Thank you for your post.

           

          I am unable to replicate the issue with your file.  I used FileMaker Pro 15.0.3 under both macOS Sierra 10.12.4 and Windows 7, and no crash occurs.  I verified the font as Calibri 12.

           

          Arial did not break PDFs in FileMaker Pro 15.0.2.  If you had font extensions in uppercase (e.g., .TTF), then there was an issue.  Changing the font extension to lowercase (e.g.,, .ttf) resolved the issue.  This issue was addressed in FileMaker Pro 15.0.3.

           

          TSGal

          FileMaker, Inc.

          • 2. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
            emory_brown

            TSGal,

             

            Thanks for looking into this so quickly.

             

            1. The steps to reproduce are as follows: Save/Send records as... PDF.
            2. The resulting PDF looks like the attached.

             

            You are correct that the font extensions caused issues, not Arial itself, however due to the IT nightmare of going through all user machines to change font extension from upper to lower case was more costly than simply using a different font on every FileMaker print form. In our particular case that involved migrating away form Arial Narrow to Calibri. This was a work-around while the issue was still under investigation.

             

            I hope this is more helpful in getting to the bottom of this. We have already pulled the trigger on simply moving fonts away from Calibri to another font which renders unrecognizable characters, since this issue needs a quick resolution.

             

            Thanks,

            • 3. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
              TSGal

              emory_brown:

               

              My apologies.  When testing your file, I confused your issue with another customer issue.

               

              I am able to replicate the issue on both macOS Sierra 10.12.4 and Windows 7.  All information, including your sample file, has been sent to our Development and Testing departments for review.  When I receive any feedback, I will let you know.

               

              TSGal

              FileMaker, Inc.

              1 of 1 people found this helpful
              • 4. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                TSGal

                emory_brown:

                 

                A tester was able to reproduce the issue using your file with FileMaker Pro 14.0.6 and FileMaker Pro 15.0.3.  The issue did not occur with FileMaker Pro 14.0.5.

                 

                The tester also noted that the issue could not be reproduced with a new file.

                 

                All information has now been sent to Development for further review.

                 

                TSGal

                FileMaker, Inc.

                • 5. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                  emory_brown

                  TSGal,

                   

                  The file I attached was a brand new file originated from FileMaker 15.0.3.

                   

                  I simply pasted a string with unrecognizable character that was causing issues. It seems to me that copying the string of data in my example file would reproduce the issue. I've confirmed this is the case.

                   

                  I've attached the new brand new file and resulting PDF. To reproduce, simply create a new file with a new field and then paste the entire data string in my example solution into that file. Change the font on the field to calibri, save as PDF, and there you go.

                   

                  This was a mac mini, running macOS Sierra.

                   

                  Hope this is helpful.

                   

                  Thanks,

                  • 6. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                    TSGal

                    emory_brown:

                     

                    I can definitely reproduce with a new file, and I have sent your NewFile.fmp12 to Testing for further review.

                     

                    The null character seems to affect Calibri font.  I will leave it to Testing to try with other fonts.

                     

                    TSGal

                    FileMaker, Inc.

                    • 7. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                      TonyWhite

                      Hi All,

                       

                      Did I hear "null character"? (6 Re-tweets and counting...)

                       

                      This may (or may not) be helpful:

                      Tony White on Twitter: "Remove #FileMaker NULL $nullChar = Base64Decode ( "AA==“ ) $cleaned = Substitute ( TO::Notes …

                       

                      $nullChar  =  Base64Decode ( "AA==“ )

                      $cleaned  = Substitute ( TO::Notes ; $nullChar ; "" )

                       

                      Tony White

                      3 of 3 people found this helpful
                      • 8. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                        emory_brown

                        Hey Tony,

                         

                        Thanks for the contribution. I verified that null character does break the font. This is likely the primary unrecognizable character which is getting input into the system.

                         

                        This is a very elegant work around, I'd been using Filter() , which still has it's merits.

                         

                        There does seem to be a deeper issue here, and the simpler work around when dealing with a large data structure of many tables with thousands of potential fields to contain null is to use fonts which work.

                         

                        Thanks,

                        • 9. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                          TonyWhite

                          We often Substitute ( $text ; $searchString ; $replaceString ) out the bad chars, rather than Filter ( $textToFilter ; $filterText )  in the good chars, as the number of possible good chars can be very large and even exceed the capacity of the calculation engine, IIRC.

                          Using Substitute can be a useful approach when working on systems that store multiple languages.

                          As an addendum to the Base64Decode note above, it seemed to work better in FileMaker 14 than 13. In our use case we were able to find and remove the char. We told the client that if the issue occurs again we will deal with it then. If it does not happen again then :-)

                          It would be good if our friends at FileMaker, Inc. filtered out the NULL and any other Unicode chars that make no sense.

                          An interesting question is how does the NULL char get into FileMaker. I have heard that Microsoft Word is a suspect.

                          Tony White

                          • 10. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                            emory_brown

                            I can say with some confidence that pasting content from Adobe Acrobat, and possibly alsoMicrosoft Outlook, are the two biggest culprits. Word may also be a suspect.

                             

                            Appreciate the advice, I went with Filter() because I wasn't sure how many other unicode characters were causing issues and I needed a quick fix. Substitute does seem like more scalable approach.

                            • 11. Re: Calibri + unrecognizable character breaks FileMaker PDF Engine
                              BobbyO'steen

                              Thanks to all for your posts and replies. This has been an issue we have run into as well. Unfortunately, Calibri is used across the majority of our solutions. Looks like we have some work ahead of us.