7 Replies Latest reply on Feb 8, 2012 7:55 AM by mbraendle

    Link checker

    Fimano

      FMS 11, FMP 11 clients, Mac OS X (10.6), no plug-ins.

       

      I have a database with some 4000 records containing data in a url field.

      One user will have the responsibility of checking these links on a regular basis.

      At a later stage, I might like to run the script on the server, but for now, I am thinking of a button that the user clicks before going to lunch...

       

      Question: How do I best go about creating a script that checks and does a summary / list of problematic links?

       

      Thanks,

      Jens Rasmussen

      Fimano

        • 1. Re: Link checker
          RubenVanDenBoogaard

          Hi Jens,

           

          You could do this is making a looping script which goes through all the 4000 records and loads the url in a (global) textfield using the 360works ScriptMaster plugin 'GetURLasText' function  or if you don't want to use plugins or prefer long lunches use a webviewer :-)

           

          Then check the textfield for errors and mark the records with errors to generate a report.

           

          Best regards,

           

          Ruben van den Boogaard

          Infomatics Software

          ruben@infomatics.nl

          • 2. Re: Link checker
            mbraendle

            You specified FMS11 in your setup.

             

            You could write a PHP script that is executed server-side and which loads the URLs.

            You can the check the HTTP status codes that are returned upon loading an URL with the PHP http_response($url) function and update your records in the database.

            • 3. Re: Link checker
              Fimano

              I was thinking of the WV approach. I know it would probably be slow. Could you be more specific about the process to parse the http response in the WV?

              Thanks,

              Jens

              • 4. Re: Link checker
                Fimano

                Very interesting. I am only familiar with running php on web pages. How would you make a php script run triggered by the FMS engine? And how to get a status of the result into the current record?

                Thanks,

                Jens

                • 5. Re: Link checker
                  RayCologon

                  fimano wrote:

                  Could you be more specific about the process to parse the http response in the WV?

                   

                  Hi Jens,

                   

                  You will first need to assign an object name to the web viewer. Once you've done that, the calculation expression:

                   

                  GetLayoutObjectAttribute("YourWebViewerObjectName"; "Content")

                   

                  ...will return the html source of the page that's loaded in the web viewer. One caveat, however, is that the function will return partial page source while the web viewer is still loading, so you will need to have the script loop and wait until the page has fully loaded.

                   

                  Regards,

                  Ray

                  ------------------------------------------------

                  R J Cologon, Ph.D.

                  FileMaker Certified Developer

                  Author, FileMaker Pro 10 Bible

                  NightWing Enterprises, Melbourne, Australia

                  http://www.nightwingenterprises.com

                  ------------------------------------------------

                  1 of 1 people found this helpful
                  • 6. Re: Link checker
                    RayCologon

                    fimano wrote:

                    Very interesting. I am only familiar with running php on web pages. How would you make a php script run triggered by the FMS engine? And how to get a status of the result into the current record?

                    Thanks,

                    Jens

                     

                    Hi Jens,

                     

                    You'll be able to call a script via PHP and have it run on the server by using the Set Web Viewer [ ] script step to load the appropriate (URL) script call into a web viewer.

                     

                    Then, using the technique I described in the previous post, you'll be able to have the script loop waiting for the result to load into the web viewer. However, be sure to build a time-out condition and error handling into the script in case anything goes awry.

                     

                    Regards,

                    Ray

                    ------------------------------------------------

                    R J Cologon, Ph.D.

                    FileMaker Certified Developer

                    Author, FileMaker Pro 10 Bible

                    NightWing Enterprises, Melbourne, Australia

                    http://www.nightwingenterprises.com

                    ------------------------------------------------

                    • 7. Re: Link checker
                      mbraendle

                      I would not use a web viewer for this task since for link checking you don't need to load the whole HTML code; way too much overhead. It's sufficient to get the HTTP response code only (which is anyway in the HTTP header, not in the web page). Most web servers will tell you anyway via HTTP response code whether the page still exists, has been moved (uses a redirect), and if and when it has been changed.

                       

                      So a simple Open URL to start the PHP script would do it, if you need to do out of an FM Pro client.

                      Otherwise, on FMS, you start in schedules a shell script that loads the PHP page with curl.

                      Or, even without FMS schedules, you can start the shell script with cron (on Mac OS X machines, I'm pretty confident there is an equivalent on MS Windows).

                       

                      fimano: To update the database records with PHP, you can use FileMaker::newEditCommand() and setField().

                       

                      As a side remark: We use you completely different approach to check about 150'000 links that we have on more than 2'500 pages with a perl program, linklint  (http://www.linklint.org). This software does two passes - it first checks the internal link structure and collects all remote links into a text file. In the second pass, it parses all the remote links. For 150'000 links it needs about 18 hours. For both passes, reports are written:

                       

                      http://www.infochembio.ethz.ch/links/urlcheck/                            (internal links)

                      http://www.infochembio.ethz.ch/links/urlcheck/urlindex.html         (remote links)

                       

                      We use this software since more than 10 years and do link checks monthly. Never experienced any problems.

                      1 of 1 people found this helpful