AnsweredAssumed Answered

Scrapping prices from a web site?

Question asked by mastmann on Jan 29, 2018
Latest reply on Jan 29, 2018 by Malcolm

I have a product database wit about 230 SKU's. My SKU's are offered on a few resellers web sites and I want to be able to run a script that scrapes the pricing information from each website and puts it into a field so I can see if the reseller is breaking our MAP guidelines.


The information on the website is not in JSON format, however, a particular phrase always appears right before the price. Here is an excerpt from the html code.

... <script>

mainItemSource = "REG";

mainSkuNum = "242338";

mainPrice = "118.0";

cartPage = false;


The phrase that preceeds the price is "mainprice =" and in the above example I want to be able to return 118.0. I've used Insert from URL to put the data into a field, but have had no luck in returning the number. I know what the length of field should be and have put that in a field called MAPLenght. I've tried Middle Characters and Middle Words combined with the Position command, but the closest I've gotten to extractng the number was returning the entire line: mainPrice = "118.0".  The formula I used for that is:

    1. Insert from URL [Off; MAPChecking::URLdata ; Product Infio::$Reseller URL]
    2. Set Field [MAPChecking::ResellerPrice; Middle ( MAPChecking::URLdata} ; Position ( MAPChecking::URLdata} ; "mainprice" ; 13+MSRP Checking::MAPLenght; 1 ) ; 13+ MSRP Checking::MAPLenght )

Once I've figured out how to return the proper number, I'll be able to write a script that cycles through the entire database checking the price for every SKU and will repeat for multiple resellers.


Thanks for any help.


Mark Astmann