1 2 Previous Next 15 Replies Latest reply on May 12, 2015 10:07 AM by TSGal

    FileMaker Server 12 crashing

    kdoronzio

      Summary

      FileMaker Server 12 crashing

      Product

      FileMaker Server

      Version

      12.05.551

      Operating system version

      Mac OS X 10.8.5

      Description of the issue

      FileMaker Server 12 crashes 1-4 times a week.
      The admin console quietly changes from running to stopped.
      Every time, the console shows:
      "DebugSymbols was unable to start a spotlight Query" then "failed lookup LS service" then "Unable to lookup coreservices" (about 5 repetitions), then "Saved crashed report for fmserverd version ???"

      We have a fairly complicated database with around 40+ users. There are many related FileMaker files and we use ESS to move data into the system from various sources (we recently stopped using ESS on the client front end in hopes of stopping the crashes. This MAY have slowed down the frequency of crashing, but did not stop it). We also use PHP web publishing to display some data.

      We have been unable to correlate the crashes with any specific events, such as large ESS imports, PHP queries, or complicated FileMaker queries. Not to say there is no correlation, just that we haven't found any yet.

      Steps to reproduce the problem

      Unable to reproduce. Happens 1-4 times a week (lately around 2 times/wk).

      Expected result

      Server continues running without issue.

      Actual result

      Server shuts itself down, and produces a crashlog.

      Exact text of any error message(s) that appear

      No FMPro messages (aside from "Unexpectedly disconnected from host").

      In console:
      "DebugSymbols was unable to start a spotlight Query" then "failed lookup LS service" then "Unable to lookup coreservices" (about 5 repetitions), then "Saved crashed report for fmserverd version ???"

      Configuration information

      - ESS
      - PHP
      - many (around 40) databases being hosted, many of which are related

      Workaround

      none.

      log.png

        • 2. Re: FileMaker Server 12 crashing
          TSGal

               kend:

               Thank you for your posts.

               I have downloaded all ten crash reports.  Except for the crash report from 26 June, the threads that crashed were centered around the SQL/ODBC connection.  The crash from 26 June appears to be related to network connection.

               I've sent all crash reports to our Development and Testing departments to get specific details about the crashes.  When I receive any feedback, I will let you know.

               TSGal
               FileMaker, Inc.

          • 3. Re: FileMaker Server 12 crashing
            kdoronzio

                 Thank you for the quick reply.

                 I have a couple of follow up questions - I don't know if the testing departments will be able to determine this - 

                 - Is it possible to tell what file &/or ESS source is causing the crash so that we can better isolate it?

                 - We haven't been able to correlate this to a specific layout or script process. Is it possible that the ESS process has already finished, and the crash happened later? We've been assuming this was an error related to memory resources only because it took a period of time between crashes. 

            • 4. Re: FileMaker Server 12 crashing
              TSGal

                   kend:

                   There is no information that I see in the crash log that shows which ESS table is being accessed or trying to connect.  Rather than speculate, I want to wait to hear from Development and/or Testing to see if it is in any way related to memory.  This also goes for the unrelated crash on 26 June.

                   TSGal
                   FileMaker, Inc.

              • 5. Re: FileMaker Server 12 crashing
                TSGal

                     kend:

                     Your crash log is similar to an issue with ESS-based Date/Time/Timestamp value lists in a drop-down list.  The reported issue only occurs under Mac OS X, and the drop-down list must not include an arrow.  This did not occur with SQL Server, but did occur with Oracle and MySQL.

                     Does this appear to be similar to your setup?

                     TSGal
                     FileMaker, Inc.

                • 6. Re: FileMaker Server 12 crashing
                  kdoronzio

                       We are all OS X.

                       We do perform searches on Timestamp fields, but we have no layouts that use a drop-down lists on ODBC sources. I just reviewed all ESS layouts and have been unable to find any that fit that description.

                       Since we stopped using ESS on the client front-end because of this issue (we're now just searching and importing ESS sources into FM tables), we also decided there's no reason to have any fields on the ESS layouts, so we removed all fields from those layouts. That was a couple of days ago and so far we haven't crashed. We'll have a pretty good idea if that fixed the issue at the end of the week.

                       Thanks

                       Ken

                        

                  • 7. Re: FileMaker Server 12 crashing
                    TSGal

                         kend:

                         Thank you for the information.

                         Please keep me updated either way....  failure or success.

                         TSGal
                         FileMaker, Inc.

                    • 8. Re: FileMaker Server 12 crashing
                      kdoronzio

                      TSGal:

                      We continue to have sporadic crashing on this server.

                      We are no longer using client-facing ESS. We are instead performing scheduled imports via ESS several times a day. 

                      Last week, the server crashed three times. Once the week before. It does seem to happen late afternoon each day, although that wasn't the case in my earlier report.

                      Does this sound like the "ESS Overload" known issue? 
                      http://forums.filemaker.com/posts/ad61a7e781?commentId=294611#294611

                      Please let me know where I can send log files - I think the address I have for you is an old one.

                      Thank you,

                      Ken

                      • 9. Re: FileMaker Server 12 crashing
                        TSGal

                        kend:

                        Thank you for the additional information.

                        I would definitely like to see the crash reports and the Log files.  Check your Inbox at the top of this page for instructions.

                        At this time, the link you provided does not appear to be related.  I won't know with certainty until I receive your crash reports and Log files.

                        TSGal
                        FileMaker, Inc.

                        • 10. Re: FileMaker Server 12 crashing
                          TSGal

                          kend:

                          I received your files.  Thank you.

                          The crash reports are almost identical to your original issue.  The crashed thread contains SQLConnect and ODBCConnection error messages, so FileMaker Server is having issues connecting to your ESS source.

                          I have sent the crash reports to our Development and Testing departments for review.  I will keep you updated as information becomes available to me.

                          TSGal
                          FileMaker, Inc.

                          • 11. Re: FileMaker Server 12 crashing
                            kdoronzio

                            Given that this is still happening consistently, and we've removed all user-facing ESS interfaces, what else can we do to minimize this? 

                                   
                            • Is this memory-related? Would restarting the server on a daily basis help?
                            •      
                            • Is there a particular ESS/ODBC connection we can identify from the logs to at least begin to contain this? Or make a decision as to the value of the ESS connection?
                            •      
                            • Would running the ESS-based routines on the server instead of a robot machine cause fewer issues?
                            •      
                            • Are there particular type of things we should avoid doing with an ESS connection? Such as GTRR, aggregate functions, etc.?

                            Thank you.

                            Ken

                            • 12. Re: FileMaker Server 12 crashing
                              TSGal

                              kend:

                              Reading the notes by Testing and Development, it appears the issue is caused by three or more types of ODBC Data Sources using the current iODBC library.  If you reduce the number of ODBC Data Sources to two (one for FileMaker Web Publishing and one for connecting to ESS), then the issue does not occur.

                              I'll continue to keep you updated as information becomes available to me.

                              TSGal
                              FileMaker, Inc.

                              • 13. Re: FileMaker Server 12 crashing
                                kdoronzio

                                Hi - I'm not sure I understand.

                                We're not using ODBC as part of web publishing.

                                For ESS:

                                We've defined a total of three ODBC DSN's on the server. One uses an Oracle source, one uses a Sybase source, and one uses a mySQL source. It's a Mac server using Actual Tech's ODBC drivers.

                                This is a large, multiple file solution. Approximately 5 of the files access at least one of these ESS sources. One or two might access 2-3 of these sources.

                                Are you saying that the issue is being caused by having one too many ODBC sources? Is there anything we can do about that?

                                Thanks

                                Ken

                                 

                                • 14. Re: FileMaker Server 12 crashing
                                  TSGal

                                  kend:

                                  My apologies for reading your crash report incorrectly.  There is reference to FMODBC-ODBCConnection, and I usually see this with external databases connecting to FileMaker Pro.

                                  You do have three ODBC DSN's on the server, so this is the current issue.  If you reduce this to two, I've been informed that this issue will no longer occur.  This is the only workaround until this limitation can be changed.

                                  TSGal
                                  FileMaker, Inc.

                                  1 2 Previous Next