11 Replies Latest reply on Aug 6, 2014 4:35 AM by beverly

    ANSI to UTF8 conversion

    ashutoshshah

      Is there any way by which we can convert an ANSI file to UTF8 format ?

        • 1. Re: ANSI to UTF8 conversion
          beverly

          What kind of file?

           

           

          -- sent from myPhone --

          Beverly Voth

          --

          • 2. Re: ANSI to UTF8 conversion
            ashutoshshah

            its an ANSI-270 format.

            • 3. Re: ANSI to UTF8 conversion
              beverly

              not the encoding the file type: TEXT file? any particular extension?

              • 4. Re: ANSI to UTF8 conversion
                ashutoshshah

                it can be any type of example. Means I will have to download the file which will have ANSI-270 type of data and further I want to read the file and convert it into human readable format i.e. UTF-8 format.

                For example.

                 

                File has following data

                     ISA*00*          *00*          *33*95216 *ZZ*TPUniqueID *140709*1547*^*00501*000000007*0*T*:~

                and I want to read the content.

                 

                Let me know if you need anything extra regarding this.

                 

                Thank you for the quick response.

                • 5. Re: ANSI to UTF8 conversion
                  beverly

                  That is just TEXT and it appears as an EDI "format". so converting the encoding won't necessarily help you. You need to PARSE the text (read it's parts) with the FileMaker functions. You must have a particular schema or map of the data, so you know what each of the sequences mean. "*" is the "field" separator and "~" is the row separator, but not necessarily the "record" separator. Each line can start with a different "code" (Segment in EDI terms)...

                   

                  <http://en.wikipedia.org/wiki/Electronic_data_interchange>

                  &

                  <http://www.webedi.co.uk/glossary.php>

                   

                  I worked with this format starting over 14 years ago (and versions of it in XML, since). You must know what the Segments and each "field" within in it mean.

                   

                  Beverly

                  • 6. Re: ANSI to UTF8 conversion
                    ashutoshshah

                    Thank you for the response.

                     

                    I am aware about the line segment and field, What I want is to get rid out of the ANSI format, so that I can read the actual content. It would be really benefitial for me if you suggest me such a parser that will parse the EDI data, since it is a very big challenge for me to devlope the same parser.

                     

                    Thank you.

                    • 7. Re: ANSI to UTF8 conversion
                      beverly

                      It's not in "ANSI format" (or encoding). The "text" that needs to go into a field is what appears between the delimiters ("*" in this case, but sometimes it's "|" for EDI).

                       

                      How are you getting the text into FileMaker?

                       

                      From there, if it's imported into a single field, you must break apart each of the "~" lines and each of the "fields" into the appropriate places. How it's done might depend on the occurrence of the return character in the text. Do you know if there is any CR (or CRLF) in the text?

                       

                      While there may or may not be a "parser" out there, it really depends on the "mapping" for your particular flavor of EDI. Do you know that?

                       

                      I use variables and loops to go "line-by-line" of the original text, then use PatternCount(), Position(), Substitute(), Left(), Middle(), Right() and other text functions in FileMaker to break apart the individual fields.

                       

                      That "MAP" is ever so important. Without it, you don't know what the fields are. Sorry I can't be more specific, but you're not giving enough information.

                       

                      Beverly

                      • 8. Re: ANSI to UTF8 conversion
                        ashutoshshah

                        Thanks Beverly for your quick response & looking into it. I really appreciate.

                        Please find the below answers for your queries.

                         

                        Do you know if there is any CR (or CRLF) in the text?

                        Nope, there is no as such line break or CR or CRLF. I can give you an exact example if needed.

                         

                        While there may or may not be a "parser" out there, it really depends on the "mapping" for your particular flavor of EDI. Do you know that?

                        Its actually in ANSI 270 & ANSI 271 format.

                         

                        That "MAP" is ever so important. Without it, you don't know what the fields are. Sorry I can't be more specific, but you're not giving enough information.

                        Please let me know if you need any specific requirement for the same.

                         

                        Please let me know if you need anything else..

                         

                        Ashutosh_S

                        • 9. Re: ANSI to UTF8 conversion
                          beverly

                          both of these are EDI formats for HIPAA. here is a full sample from

                               <http://www.cms.gov/Research-Statistics-Data-and-Systems/CMS-Information-Technology/HETSHelp/index.html?redirect=/hetshelp>

                           

                          ISA00     00     ZZSUBMITTERID ZZCMS     1408310734^005010000050141P|~

                          GSHSSUBMITTERIDCMS201408310734115014X005010X279A1~

                          ST270000000001*005010X279A1~

                          BHT002213TRANSA20140831*073411~

                          HL1201~

                          NM1PR2CMS***PICMS~

                          HL21211~

                          NM11P2IRNAME***XX1234567893~

                          HL32220~

                          TRN1TRACKNUM*ABCDEFGHIJ~

                          NM1IL1LNAMEFNAME***MI123456789A~

                          DMGD819400401~

                          DTP291RD8*20140201-20141031~

                          EQ*10^14^30^42^45^48^67^A7^AD^AE^AG^BF^BG~

                          EQ**HC|80061~

                          EQ**HC|G0117~

                          SE15000000001~

                          GE15014~

                          IEA1000005014~

                           

                          It is far too complex, without knowing your FIELDS in your database to give a complete answer. As I said you need to get the data into ONE Field. From there, you use scripts, calculations, etc. to pull each "line" (a line ends with "~") and then pull apart the fields.

                           

                          A brief example:

                               1. place the contents into one field

                               2. script loosely translated:

                                    set variable ( $all, <<field>>)

                                    set variable ( $list_all, Substitute( $all; "~"; "¶") // this makes a "return delimited" list of all the rows

                                    set variable ( $row_count, ValueCount( $list_all ) // how many rows?

                                    set variable ( $counter, 1 )

                                    loop

                                          

                            1. here we process EACH row

                                    set variable ( $this_row = middleValue($list_all; $counter; 1) ) // grab based on the position in the list

                                    set variable ( $list_row = Substitute( $this_row ; "*" ; "¶" ) //

                           

                                          

                            1. here we process EACH portion of the segment

                           

                           

                           

                                    exit loop if ( $counter = $row_count ) // stop when last row

                                    set variable ( $counter, $counter + 1) // increment the counter

                                    end loop

                           

                           

                          I did not complete the part of the segment, as this is where it gets complex. you need to test the first value in each row. This helps you know what the pattern is after that. So you may have case () function statements or call sub scripts for each segment or ??? your choice. You basically extract each part and push to variables, which then get pushed to fields (in the same record or other tables).

                           

                          AS things seem to duplicate (segments), it may depend on what it follows. You may be getting into the realm of related records here and while possible, beyond any more assistance from me.

                           

                          Beverly

                          • 10. Re: ANSI to UTF8 conversion
                            ashutoshshah

                            Hello Beverly,

                             

                            Thank you for providing such a help, It really helped me a lot, And also I am glad to see that you are having knowledge for EDI. Actually, I am starting my work on EDI, And Currently I implemented the code which you had shared yesterday and it worked for me ! ! ! ! Thank you again. And I will get back to you if I need anything beyond this.

                             

                            Thank you again.

                             

                            --

                            Shah Ashutosh H.

                            • 11. Re: ANSI to UTF8 conversion
                              beverly

                              Great! Writing these values out if you need to send them is a little

                              easier. It's just calculating and exporting. Transmitting of course is

                              usually the tricky part with EDI. But those FM text functions can be quite

                              powerful if you use them! Combine them and you can read just about any text

                              and pick it apart.