10 Replies Latest reply on Nov 17, 2016 1:38 PM by ErichWetzel

    FMS installation appears to be preventing search engines indexing custom web site

    ErichWetzel

      This is a weird one but is extremely important because our web presence has disappeared from search engines.

       

      Mac OS 10.11.5.

      FMS 15.0.1.137.

      Two machine installation.

      Both Mac Mini Servers late 2012 NOT running OS X Server.

      FMS installation used for Web direct AND Custom web sites hosted by the FMS.

       

      All worked well until we discovered that we cannot be found on the internet.

       

      My web sites have disappeared from Google/Bing/Yahoo/etc. search results.

       

      Testing the web page headers I found this: X-Robots-Tag "noindex, noarchive, nosnippet

       

      This statement tells the robots not to index or archive our pages. It shows up on every single page. We do not have statements to this effect on any of our pages because we want the search engines to index and archive all of our pages.

       

      After a day of work, found the following at the end of the Apache installation in FMS https.conf:

       

      # Globally disallow robots from crawling the sever

      <Directory />

          Header Set X-Robots-Tag "noindex, noarchive, nosnippet"

      </Directory>

       

      This seems to be setting the headers of all of the hosted web pages to not permit indexing by search engine robots. I commented out the header line  and rebooted in the hope that the system would stop saying that search engines can't index my pages.

       

      It does not appear to have stopped every header in my site coming out with the following in header tests :

       

      X-Robots-Tagnoindex, noarchive, nosnippet

       

      I do not currently have a work-around.

       

      Does anyone have any other ideas for getting this behavior to stop?

       

      I do not have this setting on in any of the actual pages. I do not use an htaccess file.

       

      I am desparate to get this problem resolved quickly since we cannot currently be found on the internet because of it.

       

      Thanks

      -Erich

        • 1. Re: FMS installation appears to be preventing search engines indexing custom web site
          ErichWetzel

          Turning off Web Direct does not change the issue.

           

          Header test result:

           

          SERVER RESPONSE: 200 OK

          Date: Thu, 28 Jul 2016 13:39:32 GMT

          Server: Apache/2.4.18 (Unix) LibreSSL/2.2.6

          Last-Modified: Thu, 28 Jul 2016 01:19:08 GMTETag: "136-538a7eba97b00"

          Content-Length: 310

          X-Frame-Options: SAMEORIGIN

          X-Robots-Tag: noindex, noarchive, nosnippet

          Keep-Alive: timeout=5, max=100

          Connection: Keep-Alive

          Content-Type: text/html

           

          I really need to get the X-Robots-Tag out of this.

          • 2. Re: FMS installation appears to be preventing search engines indexing custom web site
            ErichWetzel

            This just gets worse and worse.

             

            I reverted to FMS14. The problem went away!

             

            Filemaker: How can you sell this platform as being able to generate custom web sites but not allow them to be indexed by robots? How does that make sense?

             

            Please follow up on this. We have used your platform since the Claris days and want to help move forward. We have worked with all of your adjustments in the past. This one is very frustrating. This problem has probably damaged our primary business and a small business we were trying to get traction with.

             

            Thank you for your time. Please follow up.

            • 3. Re: FMS installation appears to be preventing search engines indexing custom web site
              TSGal

              ErichWetzel:

               

              Thank you for your post.

               

              I do not have an immediate answer.  I have sent a message to our Development and Testing departments for an explanation and workaround.  When I receive a reply, I will let you know.

               

              TSGal

              FileMaker, Inc.

              • 4. Re: FMS installation appears to be preventing search engines indexing custom web site
                ErichWetzel

                Thank you. I'm happy to help with anything I can so I can get back to FMS15.

                • 5. Re: FMS installation appears to be preventing search engines indexing custom web site
                  TSGal

                  ErichWetzel:

                   

                  Development and Testing have confirmed this was an intentional design change for security reasons to make it harder to find private WebDirect servers.

                   

                  For Mac OS X, remove the same lines from the following files:

                    httpd-fmsadminserver.conf

                    httpd-ssl.conf

                    httpd-ssl.conf.2.4

                    httpd.conf

                    httpd.conf.2.4

                   

                  For Windows, remove similar lines from:

                     web.config

                   

                  After restarting the server, it could take a few days and accessing the pages for the GoogleBot or BingBot to re-crawl the pages.

                   

                  TSGal

                  FileMaker, Inc.

                  • 6. Re: FMS installation appears to be preventing search engines indexing custom web site
                    ErichWetzel

                    TSGal,

                     

                    Thank you for the response. I will test the changes and see how we do. I do understand that it will take a couple of days for the bots to get us back into search results.

                     

                    Practical question: With the changes to FMS in recent years forcing users to use your web server rather than our own as in the past, doesn't it seem a little simplistic to refuse all indexing? What if the server is not to be strictly "private" as your tech staff put it? The software is sold as able to provide "custom web sites" not "custom web sites that cannot be indexed by search engines." My sites are not stored in the FMS library to avoid this type problem. I don't want search engines in my systems and backend either. But FileMaker set it up to refuse ANY indexing through YOUR Apache installation which I am required to use today.

                     

                    I have been with FileMaker since the Claris days and the last couple of years have been the most frustrating over the decades due to the ever increasing lack of flexibility the server package is allowing.

                     

                    The impact this has had on us and our customers makes me very angry. I cannot be the only one who uses the software in a public web site. There may be other users out there who have also had their search results evaporate since installing FMS15 and don't even know yet. While I have a willingness to edit code, because FileMaker and Apple have both forced me to get accustomed to it, many users expect all of this to just work because that is how the companies advertise it. For custom sites it does not.

                     

                    I have a "real" job here. Running our web site and systems is a "fun" sideline for me. However, I have lost three full days of real work time and many hours of personal time, that I truly did not have available to give away to this problem.

                     

                    Do you guys have a customer service person or ombudsman who would have an interest in hearing about your real users and the impact that seemingly short-sighted decisions like refusing to allow the entire contents of the FMS system to be indexed has had on us? Can you put me in touch with them?

                     

                    Thank you for taking the time to find me a way past this and taking the time to understand the impact it has had.

                     

                    -Erich

                    • 7. Re: FMS installation appears to be preventing search engines indexing custom web site
                      TSGal

                      ErichWetzel:

                       

                      First, I do understand the impact it has had on you.

                       

                      Security remains very important to us.  One reason for the change was from customers who complained about their private servers being broadcasted over the internet.

                       

                      I have proposed to Development and Testing a public/private preference when deploying.  I also recommend that you enter your comments on our Product Ideas board at:

                      Product Ideas

                       

                      The Product Ideas board is monitored by Product Management and Development.  All entries are read, discussed and considered for possible implementation in a future release.  Your suggestion will also be made public so others can  add to your input.

                       

                      TSGal

                      FileMaker, Inc.

                      • 8. Re: FMS installation appears to be preventing search engines indexing custom web site
                        ErichWetzel

                        Thank you. I will add some thoughts to the Product Ideas board.

                         

                        I understand that some do not want their Web Direct system indexed. I would agree that Web Direct content should not be searchable on the web. In that case, why not have Apache limit the indexing of anything with /fmi in the URL? My site does not go through Web Direct, it is a custom site.

                         

                        I don't know how to address it, but FileMaker still has a potential issue where there may be people out there with custom web sites under FMS15 who cannot get robot indexing and are expecting it.

                         

                        Enjoy the weekend!

                         

                        -Erich

                        • 9. Re: FMS installation appears to be preventing search engines indexing custom web site
                          TSGal

                          ErichWetzel:

                           

                          This issue has been addressed in FileMaker Server 15.0.3.

                           

                          TSGal

                          FileMaker, Inc.

                          • 10. Re: FMS installation appears to be preventing search engines indexing custom web site
                            ErichWetzel

                            TSGal,

                             

                            Excellent! Thanks for keeping me informed. I will take a look at the changes when I update.

                             

                            Erich Wetzel