1 2 Previous Next 21 Replies Latest reply on Jul 10, 2009 11:14 AM by TSGal

    FMS 10 Console crashed during scheduled backup.

    philmodjunk

      Summary

      FMS 10 Console crashed during scheduled backup.

      Description of the issue

      System: Windows server 2003. Filemaker Pro Server 10, 1 client: Filemaker Pro Adv on Windows XP system. I am testing a new solution and learning the new features of FMS 10 by running on a "test bed" of two machines on our LAN. I edited the daily backup schedule to back up to a folder on a different internal drive. (Same hardware, different partition.) The schedule was set to back up and verify all files. (9 files, 979 total MB of data.)  While the backup was running, I was doing development work on a local copy of one of the files. While the copy was local, it contained external source links to some of the hosted files. After reviewing the log this morning, it looks like the system had successfully backed up all the files and was in the process of verifying them when the following message popped up: "The admin Console cannot establish a connection to the admin server" The console then quit. No error messages appeared on the client and when I restarted the console, it showed that the scheduled back up was continuing. (Since console and services are two separate apps, I expected this.) I had to leave before the back up was complete, but all was fine when I returned in the morning. The back up copies launch normally when I copy them across the network to my client machine and open them directly with FMP 10 Adv. Good news. No apparent harm done. Bad news: my boss saw this happen and wants to know why this happened.

        • 1. Re: FMS 10 Console crashed during scheduled backup.
          philmodjunk
            

          I have now run this scheduled back up test twice and console has crashed both times.

           

          Quoting from the log:

          2009-03-12 17:00:00 Information   Starting to back up database (9 such entries one for each hosted database)

          2009-03-12 17:01:29 Warning       Administrator "fmsadmin" no longer responding; connection closed (192.168.254.9)

          2009-03-12 17:11:50 Information   Back up of .... completed (1 for each database file)

           

          and the scheduled back up continued to completion successfully.

           

          Anybody got a clue? I need to be able to reassure my boss that the new system won't crap out in the middle of business operations. Obviously, Console and server are separate apps and crashing console doesn't keep server from hosting the databases. Nevertheless an explanation as to why this happened would help lower his blood pressure and give us confidence that this upgrade is going to work for us.

          • 2. Re: FMS 10 Console crashed during scheduled backup.
            TSGal

            PhilModJunk:

             

            I can't tell you why the connection closed between the two machines.  Either the server lost connection with the console, or the other way around.  Have two machines on your network lost connection previously?  Does this happen every time you run this backup?  The log entries don't provide enough information.

             

            Once the schedule starts, information is sent to the server, and the server takes over.  It should not quit in the middle of a backup (unless of insufficient disk space on the server).

             

            TSGal

            FileMaker, Inc.

            • 3. Re: FMS 10 Console crashed during scheduled backup.
              philmodjunk
                

              I've now run this simple scheduled back up test 3 times. This same error has been waiting for me in the morning all three times so we are now 3 for 3 on this issue.

               

              You ask "Have two machines on your network lost connection previously?" Are you asking about server or client? There is exactly one computer on our network with server 10 and one with FMP 10 adv as the only client. This is our "test bed" set up for testing the newly converted database files that we soon hope to use to replace the current system. All other computers on the net have FMP 5.5 and/or FMP server (5 or 6?) and use the IPX network protocol. They thus should not have any effect on these two machines. We've seen no issues with the client workstation losing connection. The files and their back ups reside in the D: partition of the server's hard drive, which currently has 38.8 GB of free space. Not only did the scheduled backup run on Friday as expected, this is a daily schedule, so it went ahead and successfully backed up all files on Saturday and Sunday as well.

               

              Having now restarted the console, I see all files listed as "running". The database window incorrectly shows the client machine as still being connected to files it was connected to when the scheduled backup began. (After the back up started, I quit FMP on the client machine and powered it down.) I am unable to close any of the open database files and console attempts to send a disconnect message to the nonexistent client.

               

              When I examine Database Server configuration, I find:

               

              "Filemaker Server permits a maximum of 0 simultaneous FileMaker Pro connections." The "maximum number of Filemaker Pro connections" box is squeezed, showing a "2" and part of a "5". On the default folders tab, I find the "use additional database folder" check box correctly checked, the filepath shown is correct, but the status is shown as "Not a valid path". The Restore buttons are disabled; The Validate buttons are not, clicking Validate, does not change the status of the file path. The Backup Folder filepath box is blank instead of showing the file path to the back up folder.

               

              Despite this, I can use my client machine to connect to and interact with the hosted files.

               

              To "fix" the above issues, I quit console, manually stop/restart the FilemakerPro Server service. When I restart console, everything is returned to normal. I can close and open files. The Database Server configuration correctly shows that server permits 250 connections. The edit box shows 250 and correctly sized. The default folder settings for database and backup show the correct paths, their status is "Valid" and the Restore, Validate buttons are disabled unless I start to edit the file paths.

               

              There were no other applications running on the server. This is the same machine that gave us a lot of issues when we attempted to install FMserver. As described in another thread, uninstall would not work to remove FMP 5.5 and an older release of FMserver and another user (my boss) followed an IT tech's instructions for manually deleting these applications.

               

              One final observation, during the scheduled backup on Friday, I attempted to switch views from Databases to Log Viewer. Console was unresponsive.

               

              That's all the information I can see to provide. If there is anything more that you need to know, please ask. This issue will likely keep us from "going live" with the new system until it is resolved.

              • 4. Re: FMS 10 Console crashed during scheduled backup.
                philmodjunk
                  

                TS Gal,

                 

                The other thread that I refer to is labeled "Error 1330 during install of Filemaker Pro Server 10" and has never had an official "filemaker" response. If there's any chance that installation issues may be a factor, the posts in this thread may help give you a more complete picture.

                • 5. Re: FMS 10 Console crashed during scheduled backup.
                  TSGal

                  PhilModJunk:

                   

                  First of all, sorry for the late response on the other thread.  Definitely a wrong assumption on my part.

                   

                  Obviously, something was causing FileMaker Server to not display the Admin Console properly, and that is difficult to narrow down.  Now that FileMaker Server 10 is installed, there is no reason to have FileMaker Server 5/6 on the machine.  Make sure that you don't run FileMaker Server 10 and FileMaker Server 5/6 on the same machine.  This will definitely cause some problems.

                   

                  The connection I was referring to previously was not the FileMaker connection, but the system network connection between the two machines.

                   

                  If you run into any of the same problems again, please let me know.

                   

                  TSGal

                  FileMaker, Inc. 

                  • 6. Re: FMS 10 Console crashed during scheduled backup.
                    philmodjunk
                      

                    We do not run anything but FMS 10 on our server machine. We were unable to install it until we removed older versions of FMS and FMP. Uninstall would crash every time we tried to uninstall the software. We were forced to delete files manually to uninstall. Once we did that we could install FMS 10 successfully and it appears to work correctly except for the above issues.

                     

                    "If you run into any of the same problems again, please let me know"

                     

                    You did read that this scheduled back up has run three times and it has crashed three times? Given that I have made no changes to the server, I expect to see it crash again tonight.

                    • 7. Re: FMS 10 Console crashed during scheduled backup.
                      TSGal

                      PhilModJunk:

                       

                      Sorry for the confusion.  I was only making sure that FileMaker Server 5/6 was not running, and I wasn't sure if any other applications were open.  That is why I left the comment, "If you run into any of the same problems again, please let me know."

                       

                      I now have access to a Windows 2003 Server running FileMaker 10 Server.  My client machine is Windows XP, so I'm trying to set everything up to duplicate your scenario.

                       

                      I have FileMaker Server 10 installed on the Windows 2003 Server, and my client is running FileMaker Pro 10 Advanced.

                       

                      I have opened the FMServer_Sample.fp7, and a test file on the C: drive (CharacterTest.fp7).  I have set the maximum number of backups to 8, and I have set it for hourly backups to the current drive (just to make it works properly with the defaults).

                       

                      On the client machine, I have opened both files remotely and left them open overnight.

                       

                      I have 8 backups on the Server.  No crash.

                       

                      I have ODBC/JDBC turned on, even though I'm not using it.

                       

                      I tried setting the backup to the G: drive, but it aborted because I didn't have read/write privileges.  Oops.

                       

                      I then set it to D: drive, where I do have read/write privileges, and it backed up successfully.

                       

                      I'll try to find a large database file to test next.  Any other information you can provide would be appreciated.  I would like to reproduce this error.

                       

                      TSGal

                      FileMaker, Inc. 

                      • 8. Re: FMS 10 Console crashed during scheduled backup.
                        philmodjunk
                          

                        Some more info that may help narrow this down:

                         

                        1. From the Server Overview screen:
                          Web Server, PHP, Off
                          Web publishing Engine, IWP, Off; XML On; XLST On
                          Database Server, ODBC/JDBC Off
                        2. This scheduled back up is the default "daily" back up that comes with the install. I just edited it to include all the files from my secondary folder. I will be experimenting with different back up options and a differently organized file structure by grouping the files into subfolders and scheduling a back up of each subfolder at a different time. I'll report back to you what happens.
                        3. When I tried to quit FMP 10adv from the client machine (After console crash), I got lots of coffee cups. There are some simple scripts that run on some of the files when they are closed. Eventually all files were closed but the application itself did not close until I forced it to quit. (I was in a hurry and it may have gone ahead and closed if I had waited a minute or two.)
                        4. While the scheduled backup is running, console is nonresponsive. Is this normal? This time around, I was careful not to attempt interacting with console to see if that made a difference. Console still crashed.
                        5. Examining the log (It is also nonresponsive until I stop and start the service), I find the following entry: "Administrator "fmsadmin" no longer responding; connection closed. (192.168.254.9)" This entry falls after the last "starting to back up database:" entry and before the first "Backup of .... completed." entry. It occurred 2m 16 secs after the scheduled backup began. The schedule currently takes 31 minutes to complete. (one recently added file is over 2GB in size.)
                        6. There is another thread from a different user that also reports a nonresponsive console until they stop/start the service. In their case it did not occur during a scheduled back up. They also were using Windows Server 2003.
                        7.  

                          I really appreciate your efforts to help figure this one out.

                        • 9. Re: FMS 10 Console crashed during scheduled backup.
                          philmodjunk
                            

                          As promised, here's the latest:

                           

                          1. I used remove and upload to group all my files into two subfolders.
                          2. I edited the existing schedule to Back up three files (including one that is more than 2GB in size) found in the first subfolder.
                          3. I created a new schedule to back up the files in the 2nd sub folder.
                          4. I scheduled them to run at different times with several hours in between to make sure the first schedule was fully finished before starting the second.
                          5. Both schedules successfully backed up their selected files.
                          6. Both schedules caused the same console crash.

                           

                           

                          But....

                           

                          Other events were different from before.

                           

                          After the first back up was complete, I restarted console. And, there, were, no problems using console. I didn't need to restart the service to regain control of my files. There was no error warning in the log.

                           

                          During the 2nd back up,

                           

                          I got a disconnect error on my client machine for one file.

                          The client appeared unresponsive throughout the back up. (I was "coffee cupped" ).

                          I had to restart services before I could use console.

                           

                          Weird, very weird...

                           

                          More questions:

                           

                          1. According to the documentation, "FileMaker Server copies the database while it is active. Users can continue to make modifications." Yet I was coffee cupped. Is the documentation correct, or is this a new clue that something isn't right with my system?
                          2. Console is nonresponsive during the backup. Is this normal behavior?
                          3. I'm running out of new things to try here. Any body have any suggestions?
                          4. Should we attempt to reinstall the DB?
                          5. Are there some configuration files that could be purged?
                          • 10. Re: FMS 10 Console crashed during scheduled backup.
                            philmodjunk
                              

                            I think I've figured this one out. When all else fails, read the manual!

                             

                            It gradually penetrated this thick skull of mine that the server was exhibiting all the typical behavior of a machine that was just barely managing to execute the code.

                             

                            All screen refreshes were extremely slow.

                            Even simple data entry tasks on a client machine triggered a coffee cup during the back up

                            Console itself could barely manage to repond during the back up.

                             

                            So I went to the knowledge base and compared the system requirements for FM server 10 to the actual hardware I've been trying to use.

                             

                            My CPU is too slow (2.8 Ghz instead of 3.4 or higher)

                            I don't have enough RAM (only 500MB instead of 1GB w/2GB recomended)

                             

                            I suspect that once my Boss stops screaming and buys a new server, we won't have this problem.

                            • 11. Re: FMS 10 Console crashed during scheduled backup.
                              davidness
                                

                              I wonder if this issue ever really got resolved. I have the exact same issue, but we're having the problem on a brand new Dell 64-bit Windows 2008 Server computer with 8GB and 900GB of hard drive space. FileMaker Server Advanced v10.0.1.64.

                               

                              It seems that during the running of backup schedules, FMS stops being able to communicate with instances of the FileMaker Admin Console. The server continues to run fine and perform schedules, both backup schedules and server scheduled scripts. The log file continues to be written to properly.

                               

                              But the FM Admin console can no longer validate paths (and therefore can not be used to alter or create schedules), no longer displays accurate information of connected users, no longer sends out confirmation emails after the completion of a schedule, and can no longer close databases. They (FM Admin Console & FMS) stop being able to talk to one another. The console will continue to display the 'last known state' of most screens, including connected users, databases, and schedules.

                               

                              The only way to restore functionality is to restart the FileMaker Server service using the Windows Services panel.

                               

                              The loss of FM Admin Console control is, I believe, always associated with the following message in the FM Log:

                              2009-04-08 17:03:17 Server Events Warning Administrator "fmsadmin" no longer responding; connection closed. (10.10.xx.xx)

                               

                              This message always appears after a complete cycle of log entries indicating that a backup has happened; schedule is running, all of the files started and completed, oldest folder in rotation is deleted, the schedule has completed, the next backup is scheduled, than *BAM*, fmsadmin no longer responding. I assume that this next step that is failing is the transmission of some notification back to the FM Admin Console from FMS. The last initiated backup script is always perpetually shown as "Running", even though the log shows that the script has completed just fine.

                               

                              This is what I believe is happening: One (or more) of the files in the backup set contains some form of internal corruption that, although still allowing itself to be backed up, triggers some bug that stops some internal service of FMS that is responsible for communicating/updating/validating paths/something. Simply 'replacing these files' isn't as easy as you would think, as our entire system is quit large ( approx 85 files and over 8GB, soon to be 18GB, in size ). I have read all of the DevCon articles on the use of Save As Compressed/Restore/etc, and will reconstruct the databases this weekend as best as possible to see if I can resolve this. Our system is in a constant state of enhancement and revision, so I can't simply go to last month's or even last week's backup, as we've been having this problem since approximately Dec 2008, and the database structure has had too many changes to make reverting to a Nov 2008 backup of the structure feasible. User access to the databases is fine, no problems. There are just a few things that come up during the Save Compressed and/or Recover functions that makes me question whether the database structures are 100%.

                               

                              Regardless of the soundness of my databases, if the databases run fine, and back up fine, FMS shouldn't be loosing the ability to communicate and update instances of the FM Admin Console without a restart of the service.

                               

                              Any help/solutions would be greatly appreciated.

                               

                               

                              • 12. Re: FMS 10 Console crashed during scheduled backup.
                                philmodjunk
                                  
                                David Ness wrote:

                                I wonder if this issue ever really got resolved....

                                 

                                David,

                                 

                                I've run scheduled backups on the new machine (Windows Server 2003), repeatedly with no problems encountered.

                                 

                                Others are not so fortunate. There is another thread in this forum that has gotten quite long on this subject. You might try posting to that thread if you haven't already.

                                • 13. Re: FMS 10 Console crashed during scheduled backup.
                                  TSGal

                                  David Ness:

                                   

                                  Thank you for your post.

                                   

                                  There do seem to be quite a few reports of this happening, but we are unable to duplicate this internally.  Your post is well written, and I have forwarded it to the testing manager for FileMaker Server, who confirmed there have been a number of similar cases but no resolution yet.  I will post again as more information becomes available.

                                   

                                   

                                  PhilModJunk - Please send me a private message with the URL for the other link.  I'll make sure they also get notified.  Thank you.

                                   

                                   

                                  TSGal

                                  FileMaker, Inc. 

                                   

                                  • 14. Re: FMS 10 Console crashed during scheduled backup.
                                    davidness
                                      

                                    I wanted to post a follow-up on this situation.

                                    Because our FileMaker solution is quite large, at over 75 files (of varying degrees of importance), we've placed the files into a few subfolder groupings to aid in the setting up of backup schedules. Some folders get backed up every night, more important groupings; every 2 hours.

                                    The problem discussed in this thread, characterized by loss of FM Server Admin functionality, would seemingly occur during the period in which one particular backup group of 9 files was in progress. This group was backed up every two hours. Not every instance of this backup schedule would trigger the problem, but when the problem did eventually happen, it seemed to always be during this one backup schedule.

                                    Figuring that one of the files in this group itself was the cause of the problem, I broke up the group of 9 into individual folders, and created separate backup schedules for each one, figuring the problem would occur during one of these 1-file backup schedules. This would isolate the offending file.

                                    We waited.

                                    We're still waiting.

                                     

                                    After more than a week, the problem has not reoccurred. Previously, when the group of files was being backed up as a group, we would expect the problem to occur within 1 to 2 days.

                                    Therefore, I hypothesize that the problem is perhaps either with the FMS function responsible for backing up groups of files (although the group of 9 files was hardly our largest grouping in terms of number of files, although it might have been in terms of total file size). Or the problem could have been a corruption in the backup schedule itself (irrespective of the actual files being backed up), and deleting and recreating the schedule might be the reason behind the problem going away (at least so far).

                                    I wanted to add these observations in the hope that it aids either the FileMaker engineers investigating this problem, or that it might help other users who are experiencing similar problems (in which case I hope these users will post their experiences here, also).

                                     

                                    I don't consider the problem solved until I know the true cause and can reproduce the problem, but as I learn more I'll continue to post my observations to this thread.






                                    1 2 Previous Next