AnsweredAssumed Answered

Using filmmaker to scrape a public web site

Question asked by jeremy_oz on Dec 6, 2018
Latest reply on Dec 8, 2018 by fmpdude

Hi All


This may be a long shot but I am trying to get filmmaker to scrape a public website


I have had an app written in Excel by an Indian developer that does perform this function but it is a bit clunky and I have to perform a large number of searches then import the results, it all mostly works, so I know that excel can do what I need


basically it is in a number of steps


1.     There is a list of search keywords , there are actually Boats

2.     A query is placed on a public search site for boats (just like cars)

3.     A Url is returned with all of the data about the search, could be many boats fit the search criteria but typically around 20

4.     The URL return is analysed to extract specific data, make, Location, Price etc



So in a logic structure


1.     Loop through a list of search words and place a query for each one

2.     Once the URL for the return store it in a text field INSERT FROM URL

3.     Run a number of functions over the Returned data to extract the information I need and then store that in a results table.


There are two bits of this that I am struggling with


1.     How to place a query on the external web site and for example search "Outlaw"


I have placed a query manually then cut and pasted the search url into Insert From URL and the text box seems to have all of the data, then I need to write routines to extract the information I need.


So if anybody has any pointers on how to construct the initial query I would be grateful.


best regards