8 Replies Latest reply on Mar 18, 2013 1:35 PM by marksealey

    Scraping data from the web

    marksealey

      Title

      Scraping data from the web

      Post

           Hi!

           Can anyone recommend, please, a reliable way (plugin, script) to scrape data from the Web - and return it into FMP (12)?

           For instance, be able to enter a search term such as 'Renaissance painting' or 'Sculpture at the Tate' and have an FMP routine (or add-on/plugin) populate a field or fields with the resulting URLs or similar data?

           TIA…

        • 1. Re: Scraping data from the web
          philmodjunk

               Insert from URL can insert data from a web page into a text field. The specified URL text in the step can query a web site if it is constructed to correctly use syntax the web designers have specified for the web site and can include data from other text fields in your database.

               Scripts and/or calculations can then extract data from the text in that field.

               The exact details will depend on what text is inserted into the text field by the Insert From URL script step.

          • 2. Re: Scraping data from the web
            marksealey

                  

                 Thanks, PhilModJunk!
                  
                 I can see how I could use that.
                  
                 But what I really want first is some way to search before scraping… e.g. enter search terms such as 'Renaissance painting' or 'Sculpture at the Tate' - as you would with Google - and have results returned into FMP.
            • 3. Re: Scraping data from the web
              philmodjunk

                   So you would set up fields for your search terms and then the URL expression associated with Insert From URL can incorporate the data from these fields as part of the URL expression.

                   Take a look at your Contacts starter solution and note how it can map the location of a contact in a web viewer. The same method used to set up the URL expression for the web viewer can be used with Insert From URL to pull data into a text field. You may alsow want to consider using the web viewer instead of Insert From URL. GetlayoutObjectAttribute, with the "content" parameter can pull the contents of a web viewer into a text field much like Insert From Url can do the same, but Insert From URL does not use the web viewer to do so.

              • 4. Re: Scraping data from the web
                marksealey

                     Thanks again, PhilModJunk…

                     I can see how what you describe would come second - unless Insert From URL can actually perform a search similar to Google's?

                     What I may actually be looking for is a plug-in that does the searching for me: I need to be able to enter search terms such as 'Renaissance painting' or 'Sculpture at the Tate' - as you would with Google - and get results.

                Anyone used a plug-in that will do this, please?

                • 5. Re: Scraping data from the web
                  philmodjunk

                       You would enter "Renaissance painting" into a text field. Then the Insert From URL can reference a web search engine website or any other web site with a published API so that you can determine the needed syntax and the URL can incorporate both the address of the web site and the correctly formatted contents of the text field to both access a web site and submit "Renaissance painting" to it in order to query that web site.

                  • 6. Re: Scraping data from the web
                    marksealey

                         So in theory I could query Google that way?

                         Could I have FMP 'return' the top, say, 20 results from Google and populate corresponding container and/or text fields in a suitably-constructed FMP table, then, as well?

                    • 7. Re: Scraping data from the web
                      philmodjunk

                           I know that you can query Google Maps in that way. I haven't had a need to try with with Google.

                           You'd have to research any API that they may have published to find out if this is possible specifically with Google and what format the URL would have to have in order to get the results that you want.

                      • 8. Re: Scraping data from the web
                        marksealey

                             Thanks. Will do. Very helpful.

                             Anyone know of a third party plug-in that will accomplish the same thing?