AnsweredAssumed Answered

FMS installation appears to be preventing search engines indexing custom web site

Question asked by ErichWetzel on Jul 28, 2016
Latest reply on Nov 17, 2016 by ErichWetzel

This is a weird one but is extremely important because our web presence has disappeared from search engines.


Mac OS 10.11.5.


Two machine installation.

Both Mac Mini Servers late 2012 NOT running OS X Server.

FMS installation used for Web direct AND Custom web sites hosted by the FMS.


All worked well until we discovered that we cannot be found on the internet.


My web sites have disappeared from Google/Bing/Yahoo/etc. search results.


Testing the web page headers I found this: X-Robots-Tag "noindex, noarchive, nosnippet


This statement tells the robots not to index or archive our pages. It shows up on every single page. We do not have statements to this effect on any of our pages because we want the search engines to index and archive all of our pages.


After a day of work, found the following at the end of the Apache installation in FMS https.conf:


# Globally disallow robots from crawling the sever

<Directory />

    Header Set X-Robots-Tag "noindex, noarchive, nosnippet"



This seems to be setting the headers of all of the hosted web pages to not permit indexing by search engine robots. I commented out the header line  and rebooted in the hope that the system would stop saying that search engines can't index my pages.


It does not appear to have stopped every header in my site coming out with the following in header tests :


X-Robots-Tagnoindex, noarchive, nosnippet


I do not currently have a work-around.


Does anyone have any other ideas for getting this behavior to stop?


I do not have this setting on in any of the actual pages. I do not use an htaccess file.


I am desparate to get this problem resolved quickly since we cannot currently be found on the internet because of it.