14 Replies Latest reply on Dec 11, 2013 1:28 AM by GiancarloMeak

    Parsing XML from the DDR

    DanielShanahan

      I'm using FileMaker 12 to parse data from the DDR's XML. It works well but seems slow (relative term, I know). I'm investigating parsing it in php using the simplexml_load_file () function. I'm doing this in MAMP on my local machine. I've hit some roadblocks early in the process. I think the php function either can't find the file or doesn't understand the FMPXML format.

       

       

      Before I run too far down this rabbit hole, has anyone else successfully parsed the DDR's XML in any other language (e.g. Java, Python, PHP, Javascript, etc.)?

       

       

      Thanks.

        • 1. Re: Parsing XML from the DDR
          steve_ssh

          Hi Daniel,

           

          A long time ago, just for fun, I used to write my own parsers for the FM11 DDRs.  This was very much an amateur endeavor on my part, just to challenge myself and see how far I could get.

           

          One time I gave myself the constraint of not using any plugins, and only using command-line tools available on OSX, and, if I recall correctly, the result was was:

           

            FM  ==> AppleScript => Shell script  ==> much use of awk and some use of iconv  -  and perhaps one other utility whose name escapes me right now.

           

          This actually worked nicely for my purposes, but was definitely a humble and amateur hack, and not scalable in terms of being easy to scale for v12

           

          (Note: Scalability wasn't an issue as far as the DDR file size was concerned -- it worked fine on some rather large DDRs).

           

           

          The second time that I did this, I wrote my own parsing code in Java (JDK 1.4 -- I believe).  This was off to a promising start, but, three major java versions later, it is still sitting around somewhere waiting for me to finish it...

           

           

          The pertinent points that I recall:

           

             1)  I believe that the DDR XML output was encoded in UTF-16, which came as a surprise to me, as I would have expected UTF-8.

           

             2)  I needed to included a routine that scrubbed the DDR XML for invalid characters that were sometimes included in the data, oftentimes the result of someone having named a script using some sort of upper-ASCII type character.

           

             3)  Due to the extremely large file sizes of the DDRs that I was parsing, a DOM style parser seemed out of the question (too much memory usage) -- the code I wrote utilized "event-driven" parsing.

           

             4)  It was a lot of fun, but at the end of the day, doing this for anything other than the pleasure of the challenge wouldn't make sense to me when something as great as Base Elements exists.

           

           

           

          Additionally, on a related note:

           

          There was a time when we were doing a lot of schema/script cleanup at my job, and we used Base Elements for this purpose, but our Base Elements import process took at least 6+ hours, which meant that we couldn't really afford to have "on demand" schema analysis.  Because of this, I wrote a tool which I nicknamed "DDR Inhaler" which would plow through a stack of DDR files, using strategically composed grep commands, and in a much shorter amount of time it would return a list of all elements which appeared to have missing or broken references as well as any element that contained a string reference that we used to mark elements that we aimed to deprecate/delete.  This tool did nothing in the way of determining dependancies, it only returned a list of suspicious elements and their file locations, and left the rest up to the developer.  Having this tool for taking a "quick look", combined with the omniscience of Base Elements proved to be a fairly effective combination for getting this kind of work done.

           

          Kind regards, and good luck!

           

          -steve

          • 2. Re: Parsing XML from the DDR
            wimdecorte

            Rfm has done it in Ruby,  PyMaker in Python, fmDotNet in .NET (www.fmdotnet.org), and I believe there still is a Flex one out there too...

            • 3. Re: Parsing XML from the DDR
              beverly

              Using XSLT, of course, it works well!

               

              Beverly

              • 4. Re: Parsing XML from the DDR
                GiancarloMeak

                Before investing time on understanding DDR XML if you aren't an XML XSLT guru like Beverly (take a look to her good book on XML and FMP - Beverly, XSLT for DDR could be a great Christmas present for every one! :-) I recommend that you look at Goya Base Elements.

                http://www.goya.com.au/baseelements

                I have my ways to use DDR but are absolutely not comparable with Base Elements and I've lost a lot of time on them.

                • 5. Re: Parsing XML from the DDR
                  beverly

                  Yes, BaseElements has done the work for you!

                   

                  But I guess the question goes to OP: what are you parsing it for? Isn't creation (and output as HTML) enough for your needs? Do you think you'll get something out of the XML that isn't in the HTML (or via Base Elements)? I create XSLT, to get specific bits. I have a sample that brings in the Relationships so that the records/fields are somewhat similar to the old relationship listing (remember those, before the graph?) Some people seem to need both: the visual graph and the list. I did the same thing with Scripts. It all depended on what needed to be "extracted".

                   

                  Beverly

                  • 6. Re: Parsing XML from the DDR
                    pminich

                    I am just about to embark on a project that will be importing xml data from a pos system. I was going to look for some guidance in this area.  I found a reference to Beverly's book for FMP 6. Is this the most current version? If so is there anything else to be aware of?

                    • 7. Re: Parsing XML from the DDR
                      DanielShanahan

                      Thanks all.

                       

                      Steve said it best above when he wrote:

                      A long time ago, just for fun, I used to write my own parsers for the FM11 DDRs.  This was very much an amateur endeavor on my part, just to challenge myself and see how far I could get.

                      I've created a parser in FMP that grabs data out of the DDR's XML file.  I take the result of the parsed file and use it with a Javascript data table.  It is an interesting project and I'm learning a great deal.  Plus, I have more control over what I want to see and how I want to view it.  I use the Javascript data table to search script steps as well as all calculations (e.g. field definitions, security, conditional formatting, etc.).

                       

                      I also use the DDR's HTML file when analyzing a FMP file.  I've used (and liked) Inspector Pro in the past and I have only heard good things about BaseElements.  Yet, I like the idea of creating something myself.

                       

                      I remember reading that XSLT was deprecated for FileMaker Server (http://help.filemaker.com/app/answers/detail/a_id/7701).  Not knowing much about XSLT, I thought this meant it was no longer an option for working with FMPXML.  I suspect I was wrong in that assumption.

                      • 8. Re: Parsing XML from the DDR
                        beverly

                        Hey! The IMPORT & EXPORT is exactly the same. Toss out the web publishing chapter(s), spruce it up with new screen shots. Et Voila! You now have a book on FMP13 and XML/XSLT.

                         

                         

                         

                        Not many books are 10 yrs+ can say that!

                         

                        -- sent from my iPhone4 --

                        Beverly Voth

                        --

                        • 9. Re: Parsing XML from the DDR
                          beverly

                          That was XSLT as web publishing option. Yes gone away. Import/Export with XML/XSLT? Still there, still the same.

                           

                          -- sent from my iPhone4 --

                          Beverly Voth

                          --

                          • 10. Re: Parsing XML from the DDR
                            pminich

                            Thanks!

                            • 11. Re: Parsing XML from the DDR
                              steve_ssh

                              Daniel,

                               

                              Good for you!

                               

                              I totally get how much fun this sort of thing can be, and a great learning experience as well.

                               

                              I wish you lots of fun - I bet you are going to have a blast.

                               

                              Very best,

                               

                              - steve

                              • 12. Re: Parsing XML from the DDR
                                mikebeargie

                                Sometime's other people's abilities to do it better are good enough for me.

                                 

                                I'm a big fan of beezwax inspector:

                                http://www.beezwax.net/products/inspectorPro/

                                 

                                Anything I ever needed from the data generated by DDRs

                                 

                                As for your PHP issue, the declaration (or lack of a proper one thereof) in the DDR XML might cause some issues with PHP packages. If the simplexml package came with a sample XML file, try dropping that in the same directory and naming it similar to see if you can load that file. That might at least tell you if it's the format. Or in reverse of that, copy the file to a known public directory, and try accessing it from there.

                                • 13. Re: Parsing XML from the DDR
                                  GiancarloMeak

                                  I confirm that the xml import/export features are the same so "FileMaker Pro 6 Developer's Guide to XML/XSL" it's absolutely a very useful reading also after more than 10 yrs.

                                  (And it's a cool book to have in a FMP books library :-)

                                   

                                  http://www.amazon.com/FileMaker-Developers-Guide-Wordware-Library/dp/155622043X/ref=sr_1_12

                                   

                                  Giancarlo

                                  • 14. Re: Parsing XML from the DDR
                                    GiancarloMeak

                                    Thanks Beverly, at the moment I use the HTML of the DDR, when I need to extract something (like field definitions, formulas..) I right click the html, inspect the table, copy the raw html, paste it in Open Office (or excel) and if necessary I import the XLS generated file in FMP.

                                    It's seem complex but it works quite fast for me (forgive me not using any more XSLT!)

                                    I've used Base Elements during a couple of weeks to analyze a complex solution and the power of what you can search and explore inside DDR it's amazing.