9 Replies Latest reply on Sep 12, 2013 11:39 AM by mikebeargie

    Chinese & English... then the world

    LyndsayHowarth

      Hi All,

       

      I would like to know a bit more about the ins and outs of dealing with:

      a. bilingual (Chinese & English) office

      b. clients who need to be managed in their own language.

       

      Sure I can use Chinese fonts and translation tables for database labels etc and I have done a bit of in-page google translate stuff with PHP... but other than that I have gaps in my understandings.

       

      The first problem I have encountered is that a CSV file needs to be exported for a distribution house. Whilst I pasted in chinese characters... the export replaced it with ascii characters once opened in Excel.

      Perhaps this is not an issue if my OS were Chinese... but then I would also have issues with Spanish and Greek and Thai etc...

       

      Any advise which could help me understand the issues are welcome.

       

      I am guessing there will eventually be a bit of consultancy $s for FM-language expertise that I will be putting in the budget if I get the job...

      What I want to know now is what my options are...

       

      Thankyou... in anticipation ;-)

      - Lyndsay

        • 1. Re: Chinese & English... then the world
          mikebeargie

          Regarding the CSV issue, I remember having gotten around something similar to this by exporting a single field of records (the record was one line inside the field I was exporting), and then renaming it to .csv AFTER export via a send event script step (via the MS-DOS REN/RENAME commands).

           

          Regarding dealing with a bilingual office, if it's not a language I'm comfortable parsing an e-dictionary result out of, my first step would be to hire a translator to work with me and build the interfaces out.

           

          I built a bi-lingual english/japanese interface one time that gave the user a preference option to select which language they wanted. Then there were different layouts and a navigation script that directed appropriately.

          1 of 1 people found this helpful
          • 2. Re: Chinese & English... then the world
            LyndsayHowarth

            Thanks Mike...

            I am dealing with a translator who is comfortable entering all the equivilent values for my translation table.

            I will checkout your renaming idea a bit more but any of the other formats I tried exported ascii.

             

            What I am sitting here wondering also... is what happens if a machine configured in Chinese and another configured with English... what would be the result if they connected to the same server version of FM?

             

            - Lyndsay

            • 3. Re: Chinese & English... then the world
              steve_ssh

              Hi Lyndsay,

               

              I'd like to see if I correctly understand what you are describing -- I'm not sure that I do.  I'll do my best to restate the scenario I think you are describing, just to see if I understand correctly.  I've then got a little bit of information to toss out there, which I believe may be related.  It may be stuff that you already know and have tried -- my apologies to you if that is the case.  If you haven't considered this info yet, then I sincerely hope that it may be helpful to you.

               

              Kind regards,

               

              -steve

               

               

               

              Would this be an accurate restatement of the scenario in question?:

               

                 1) You have an FM database (presumably v11 or v12 ?)

               

                 2) In at least one record, there is a field where you have pasted in some text which is in Chinese

               

                 3) The text displays fine in that field when using an FM client (FMP, FMPA)

               

                 4) You then do a CSV export from Filemaker which includes the Chinese text data

               

                 5) When you open the resultant CSV file in Excel, instead of seeing the Chinese characters, you see ascii (presumably some garbled/munged-looking text)

               

               

              Years ago I used to maintain a system that was required to shuffle international customer data between various back-end services, and anytime there was a problem where, for example, Japanese characters were being munged into something else, it was my job to track down the problem and demonstrate exactly what was going wrong.

               

              Without exception, the cause was always one of the following two explanations:

               

                 1) We were using a system/technology that was not was not sufficiently modern to support a full palate of international characters.

               

                 2) When data was being exchanged between two systems, there was a disagreement regarding the underlying charset encoding used to encode the data, i.e. System A would output data encoded using Charset X, but then System B would access that data believing that Charset Y had been used as an encoding, and under many circumstances, the problem goes undetected, but as soon as something like Japanese chars were being transmitted, the problem would become apparent, as System B would show the data munged into something else.

               

               

              Regarding Explanation #2 Above:

               

              Most of the time, the particulars of the problem were that System A would be outputting data using UTF-8, but System B would be reading the data as though it were encoded in some other charset (on Macs this would typically be the MacRoman charset, on Windows machines I believe CodePage 1252 was the typical mistaken encoding).

               

              The solution was almost always to simply employ whatever means were available to ensure that System A encoded its output using UTF-8, and System B reads the data as UTF-8.  Depending on the particular means that the data was being transmitted from system to system, this might have meant including some special flag or tag that explicitly stated which charset was being used to transmit the data.

               

               

              In the case of your FM export:

               

              The first thing I would check would be to make sure that the FM export is using "Unicode UTF-8" (set in the dialog where you specify which fields to export).

               

              The next thing that I would do is to open the file using an app where you can specify the charset that should be used to interpret the data.  On a Mac, just for testing purposes, I actually just TextEdit a lot of times for this sort of thing, because the open file dialog allows you to choose the charset for opening the file.  TextWrangler also seems to have the same option easily available in its open dialog.   If all things are working as they should, you should be able to tell either of these apps to open your CSV file using UTF-8, and you should see your data there, looking exactly as you expect it to, regardless of the default language associated with your machine.   If not, the problem really is mysterious.  (As an extra note, you can also deliberately choose the "wrong" encoding to open the file in the text editor to see if you can match the bad data that you see in Excel.)

               

              If the above goes well, and you are able to open the CSV file using one of the above text editors, and see your data as it should be, then, the next thing would be to figure out what it takes to tell Excel to open the file using UTF-8, as that is likely the problem.  A minute ago I ran the simplest of tests like that over here, and not having Excel, the CSV file on my machine gets opened by NeoOffice.  NeoOffice dutifully asked me which charset encoding to use to open the file.  I selected UTF-8, and all was good:  when I viewed the contents in NeoOffice, I saw all the "international" chars that I had entered in FileMaker without any issues.

              1 of 1 people found this helpful
              • 4. Re: Chinese & English... then the world
                LyndsayHowarth

                Thanks Steve... and sorry for the slow response.

                 

                I've had similar thoughts myself but not yet had a chance to test it. My test would have involved exporting as html with UTF-8 encoding.

                 

                I am working on a brand new iMac using Excel 2008 and FMPA 11 (although just about to try it in 12) and exporting as UTF-8.

                 

                In excel I get no options for opening a CSV with specific format and the text is ascii garbage.

                 

                I will run a few more tests... when I get a moment... and report back.

                 

                - Lyndsay

                • 5. Re: Chinese & English... then the world
                  steve_ssh

                  Hi Lyndsay,

                   

                  Here is a thread that furthers my current suspcions, as well as makes mention of Mike's technique (or something similar) of renaming the target file:

                   

                  http://superuser.com/questions/280603/how-to-set-character-encoding-when-opening-excel

                   

                  I strongly suspect that FMP is doing the "right" thing for you, i.e. properly encoding the data per your export preference.

                   

                  I believe the only real issue is getting Excel to read the file as UTF-8 instead of some other charset encoding that it assumes your CSV document to be in (ascii?).

                   

                  The couple of threads that I have just now seen suggest that there is an "import" option in Excel which is more cumbersome than using an open file command, but apparently the import option does allow one to specify the proper charset encoding to use.  I also like your idea of exporting as HTML, since an HTML document format would allow you to include some meta data to specify the charset-encoding, thereby giving Excel a cue to "do the right thing".

                   

                  I'd definitely love to hear what you find out and decide upon.

                   

                  Very best,

                   

                  -steve

                  • 6. Re: Chinese & English... then the world
                    LyndsayHowarth

                    Mike and Steve,

                     

                    Thanks to you both for the discussion.

                     

                    FileMaker does faithfully export the UTF-8 characters so mixes languages appropriately.

                     

                    Excel fails to import from any format and retain the characters... other than when saved as an html table. When using a File>open on the html file, Excel offers the UTF-8 importing option and faithfully reproduces the characters. It can then be saved as XLS and will thereafter retain the characters.

                     

                    I also tried Open Office... which encoded all the text in chinese characters with a CSV and omitted the characters completely when using a TAB / txt file.

                     

                    Text Edit read them all.

                     

                    Thanks to you both for the discussion and helping me get to this resolution. I have used this methodology before to generate a MS Word file but foreign language using non-roman characters made me think it was going to be more difficult to resolve.

                    Now if you will excuse me... I am going to mark this response as the Correct Answer although I do believe it was a combined effort. I want to know for sure if when you mark your own post as correct that it doesn't give you the 4 points... and it will help others looking for the answer by not having to read the whole thread...
                    - Lyndsay

                     

                    Message was edited by: Lyndsay Howarth   YEP...I only got 1 point :-)

                    • 7. Re: Chinese & English... then the world
                      mikebeargie

                      hah! not like we're point hounds or anything, just glad to help!

                      • 8. Re: Chinese & English... then the world
                        LyndsayHowarth

                        But who would I have chosen, anyway? The discussion lead to the answer. Your rename comment made me think of HTML and Steve's contribution was confirming UTF-8 as the appropriate encoding told the rest of the story.

                        Points are good... but I was more interested in the mechanism of distribution as I have been working on such a beast. I want to build a dual system so the points can reset every year but maintain the status level gained.

                         

                        Isn't FM wonderful? There is always more to learn!

                         

                        Sent from my iPad

                        Lyndsay Howarth

                        11th Hour Group Pty Ltd

                        • 9. Re: Chinese & English... then the world
                          mikebeargie

                          Hence my other thread that had overtones of "community values" and such. It sure is wonderful!