This is a weird one but is extremely important because our web presence has disappeared from search engines.
Mac OS 10.11.5.
Two machine installation.
Both Mac Mini Servers late 2012 NOT running OS X Server.
FMS installation used for Web direct AND Custom web sites hosted by the FMS.
All worked well until we discovered that we cannot be found on the internet.
My web sites have disappeared from Google/Bing/Yahoo/etc. search results.
Testing the web page headers I found this: X-Robots-Tag "noindex, noarchive, nosnippet
This statement tells the robots not to index or archive our pages. It shows up on every single page. We do not have statements to this effect on any of our pages because we want the search engines to index and archive all of our pages.
After a day of work, found the following at the end of the Apache installation in FMS https.conf:
# Globally disallow robots from crawling the sever
Header Set X-Robots-Tag "noindex, noarchive, nosnippet"
This seems to be setting the headers of all of the hosted web pages to not permit indexing by search engine robots. I commented out the header line and rebooted in the hope that the system would stop saying that search engines can't index my pages.
It does not appear to have stopped every header in my site coming out with the following in header tests :
|X-Robots-Tag||noindex, noarchive, nosnippet|
I do not currently have a work-around.
Does anyone have any other ideas for getting this behavior to stop?
I do not have this setting on in any of the actual pages. I do not use an htaccess file.
I am desparate to get this problem resolved quickly since we cannot currently be found on the internet because of it.