1 2 Previous Next 15 Replies Latest reply on Feb 22, 2017 7:45 AM by TSGal

    Can't add a field on a hosted file anymore, FMS hangs

    Vincent_L

      Hi,

       

      Since the beginning of this year, I'm almost never able anymore to fields to table. (As I'm writ

      Yes, I now have more scheduled scripts, and yes it can happen when a server side script is running. But, I also encountered the issue, when no server side were running.

      It happened with FMS 14 + Mac OS 10.9, and now with FMS 15v2 + Sierra.

      So now, the only way I can edit the database fields, is to disconnect all my clients, and wait a bit, then after I can add fields.

       

      You may say that it's not recommend, but in the past I never encountered this, and the real reason I'm posting this is that the behavior is not logical.

       

      Here's what happens :

       

      after editing the field definition (and sometimes just relationships), I click the ok button.

      - The progress bar fill ups rapidly to 90% but then remain stuck for ages (as I'm writing this, it's been 45 minutes)

      - Also the Mac GUI is messed up by Filemaker : Filemaker remains frontmost no matter what (and even you can get the progress modal dialog go beneath the fields window.

       

      - But both FM Client CPU usage AND FMS Cpu usage is next to nothing 0,1% !

      So basically there's nothing going on.

      - Moreover, if a server side script was running it's hanged and also no cpu usage on the fmased process

      - new user can't connect anymore to the server.

      - Backups fails with error 642,729, 8003 and 702

       

      The only thing to do is to kill the editing client, but it's not quick, and then the server remains in that idle states, server side script don't resume, new clients still can't connect anymore.

      Of course, this often corrupts the edited file !

       

      Oh, and also, after disconnecting the client (and quitting) FMS admin webpage AND fmsadmin command line still report it as connected

       

      Here are samples of the editing client, and fmserverd and fmased

       

      Dropbox - samples.zip

       

      This seriously hinders my productivity as an in house dev, and sometimes makes me restart the server during production.

       

      This reminds me the kind of "DOS attack" that occurs when cancelling unstored finds

       

      Please fix this flaw that makes FMS not a robust software

       

      Thanks

        • 1. Re: Can't add a field on a hosted file anymore, FMS hangs
          TSGal

          Vincent_L:

           

          Thank you for your post.

           

          Backups will definitely not complete if you are changing the schema.  The backup errors are normal.  Error 642 is backup aborted due to unknown error.  Error 729 is Backup aborted due to user or error.  Error 8003 is also an unexpected error during backup.  Error 702 is a backup error due to lock conflicts (since you are modifying the schema).

           

          If you are changing the schema while another user has a record locked, then FileMaker will wait for the record to become unlocked before progressing.  That is why you see the CPU at 0. 1%.

           

          Although these are expected, the issue appears to be with the client remaining connected after being disconnected.  If you send a message to the client and the client complies and disconnects herself/himself, does the issue still occur?

           

          TSGal

          FileMaker, Inc.

          • 2. Re: Can't add a field on a hosted file anymore, FMS hangs
            Vincent_L

            Hi TSGal, thanks for your answer

            I understand about locking, but something should happen, here nothing happens. Changing the schema should lock, that's ok, then the other clients, scripts should pause, and the schema editing should progress.

            Here everything is waiting one another so this can go on forever. That situation shouldn't exist.

             

             

             

            If you are changing the schema while another user has a record locked, then FileMaker will wait for the record to become unlocked before progressing. That is why you see the CPU at 0. 1%.

            • 3. Re: Can't add a field on a hosted file anymore, FMS hangs
              TSGal

              Vincent_L:

               

              What do you suggest?  Disconnecting all users when it comes to saving changes to the Schema?  A message in Manage Database saying "Users are currently connected to the database."?  A time-out after one minute without saving the schema changes?  An option to send a message to all users to disconnect?

               

              TSGal

              FileMaker, Inc.

              • 4. Re: Can't add a field on a hosted file anymore, FMS hangs
                Vincent_L

                Probably the message in Manage Database saying "Users are currently connected to the database."?

                would be a quick workaround

                 

                AND, the timeout because between the time when clicking ok and the time of completion a lock could also happen.

                 

                actually I suggested something like that here Better Live development tools but the above TWO would be perheps enough/better

                 

                But also, I'd suggest engineers should look into that because I think the issue is more prevalent than in the past

                 

                Also what's strange is that the table I edited today, is one that is never written by clients or scripts, only read, so other user / script could not lock it.

                 

                So what seems wrong to me is that the the one thing that created the lock, the editing, seems to wait for the lock to end, but it's up to it. So it's basically waiting for itself.

                 

                Also, what's not normal is that when disconnecting the editing client, things don't resume to normal (server side remains stuck, new client can't connect). So there's definitely stuff going on that's not working as designed. The server should at least be robust, and resume perfectly after disconnecting.

                • 5. Re: Can't add a field on a hosted file anymore, FMS hangs
                  Vincent_L

                  And also, especially when there's no CPU usage, there's no reason Filemaker hijacks the Mac's GUI by sticking frontmost which is a problem that I reported there Filemaker blocking Mac OS X GUI  with high cpu usage (but here it also happens with low cpu usage)

                  • 6. Re: Can't add a field on a hosted file anymore, FMS hangs
                    Vincent_L

                    Hi TSGal,

                     

                    I'm pretty sure there's a infinite locking situation issue in FMS 14/15.

                    Today I added one relationship on my main table, just created nothing else, not used anywhere. It's been 1 hour and my FMPa client is now stuck, but also my FMS serverside script are note crunching anymore as fmased and fmserverd are only using 2-3% CPU.

                    Backups are failing due to locking issue.

                    But too me there's no reason it should lock, i just added a new  relationship that's not used anywhere.

                     

                    Furthermore, even if locking would be legit, that's not a reason why nothing should happen : I can understand that if a user is editing a record, the indexing could be stuck. But here we're in full automated situation.

                    My client that is altering the schema, can of course pause the server side scripts and backups, but at least it should progress, while the other things are paused.

                     

                    But here the situation is different, it's as if we had 3 user, one is locking a record, the other 2 are stuck, but the first one is going to lunch. Here My FMP Client altering the schema is going out to lunch.

                    • 7. Re: Can't add a field on a hosted file anymore, FMS hangs
                      Vincent_L

                      so, just after writing the above post, I had to leave work, so I killed my client from the server. As previous experience showed, I was convinced that nothing (not the server side script, nor the backups) would resume.

                      Here I'm 2 hours + later, nothing resumed.

                      So I had to close all the files and restart the server.

                       

                      This is not normal, at the very least, killing the client that created the issue, and the locking cause, should have removed the locking, and hence the "paused" task should have resumed.

                       

                      And by the way, the edited file, though "cleanly" closed by the server, needed a consistency check…

                       

                      • 8. Re: Can't add a field on a hosted file anymore, FMS hangs
                        TSGal

                        Vincent_L:

                         

                        Thanks for all of the information.

                         

                        Although I cannot replicate the issue (FileMaker Server 15.0.3 running on Mac OS X 10.11.6, FileMaker Pro Advanced 15.0.2 running on macOS Sierra 10.12.1, and FileMaker Pro Advanced 15.0.2 running on Mac OS X 10.10.5), I have sent all information to our Development and Testing departments for review.  When I receive any feedback or guidance, I will let you know.

                         

                        TSGal

                        FileMaker, Inc.

                        • 9. Re: Can't add a field on a hosted file anymore, FMS hangs
                          TSGal

                          Vincent_L:

                           

                          Development has looked over the two FileMaker Server process samples.  In both cases, it looks like there are about nine client threads blocked due to a record lock or table lock.

                           

                          In the FileMaker Pro samples, both show that the schema commit is in progress, and FileMaker Pro is waiting on an upgrade of the remote lock in order to continue with the commit.

                           

                             50168 FMDefineDatabaseDialog::ConfirmOK()  (in FileMaker Pro) + 97  [0x109a600ef]

                               50168 Draco::DBSchema::Commit(Draco::InteractiveErrorHandler*)  (in DBEngine) + 270  [0x10db41fbc]

                                 50168 Draco::DBSchemaCatalog::Commit(bool, Draco::InteractiveErrorHandler*)  (in DBEngine) + 17  [0x10da72331]

                                   50168 Draco::DBCatalog::Commit(bool, Draco::InteractiveErrorHandler*)  (in DBEngine) + 292  [0x10da6dc5e]

                                     50168 Draco::DBCatalog::DoCommit(Draco::DBCatalogCommon::TransferMode, bool)  (in DBEngine) + 90  [0x10da6a122]

                                       50168 Draco::DBCatalog::Upload(bool, Draco::DBCatalogCommon::TransferMode)  (in DBEngine) + 853  [0x10da696c7]

                                         50168 Draco::DBLockMgr::Upgrade(Draco::DBLock&, Draco::DBLockType, unsigned int)  (in DBEngine) + 524  [0x10daeee48]

                                           50168 Draco::DBRemoteUpgrade::DoUpgrade(Draco::DBLockType, Draco::DBLockType, Draco::DBLockUser const&, Draco::RefCountedObjPtr<Draco::DBLockPathObj> const&, unsigned int)  (in DBEngine) + 414  [0x10daef094]

                                             50166 Draco::RCTransaction::Send()  (in DBEngine) + 1627  [0x10dc0a4d3]

                           

                          In both cases, there is a client thread on the Server that is trying to do the lock upgrade, matching what the client is doing.

                           

                          In the November 4th sample of fmsased (Server-side scripting), there are actually two scripts running, both trying to open a file as part of a Set Field script step, and both are waiting on a download from the host in order to complete the open.

                           

                          Development would like a clone of the database file(s) with instructions on how to reproduce the hang.  Specifically, which fields in which tables were modified.

                           

                          When the hang is occurring, Development would also like to get two separate samples of the fmserverd and fmsased processes about 60 seconds apart.  The best way to do this is via the command line, and force the output to be written to separate files.  For example:

                          $ sudo sample fmserverd >fmssamp1.txt

                          $ sudo sample fmsased >fmsasesamp1.txt

                           

                          $ sudo sample fmserverd >fmssamp2.txt

                          $ sudo sample fmsased >fmsasesamp2.txt

                           

                          I have sent you a private message with instructions where to send these files.

                           

                          TSGal

                          FileMaker, Inc.

                          • 10. Re: Can't add a field on a hosted file anymore, FMS hangs
                            Vincent_L

                            Hi TSGal,

                             

                            Many thanks for the followup. I think that sending you a clone won't help much: it's a very complex solution with 110 files, and files that are from 1 to 10 Gb each.It's also has lots of server side script. But more importantly, it has to have a large dataset. Rebuilding a coherent dataset from the clone wont be possible, and finally even the clone contains some confidential data like hardcoded password.

                             

                            But I'll definitely do several samples whenever this occurs (but I try to avoid this because it hampers my production).

                             

                            Development has looked over the two FileMaker Server process samples. In both cases, it looks like there are about nine client threads blocked due to a record lock or table lock.

                             

                            In the FileMaker Pro samples, both show that the schema commit is in progress, and FileMaker Pro is waiting on an upgrade of the remote lock in order to continue with the commit.

                             

                            In both cases, there is a client thread on the Server that is trying to do the lock upgrade, matching what the client is doing.

                             

                            In the November 4th sample of fmsased (Server-side scripting), there are actually two scripts running, both trying to open a file as part of a Set Field script step, and both are waiting on a download from the host in order to complete the open.

                             

                            My Feeling is that it's a self inflicted pain, the schema commit actually creates the lock, and is waiting for the lock it created to be unlocked, or that it creates a lock, which in turns locks some other clients that may create a lock themselves due to that, and then everybody is waiting for each other lock to be released.

                             

                            Also, do keep in mind that killing the schema commit (by disconnecting the source client), doesn't resume the situation.

                             

                            Development would like a clone of the database file(s) with instructions on how to reproduce the hang. Specifically, which fields in which tables were modified.

                             

                            Instructions are : just do a schema change on a big table whenever there's clients connected or server side script running. This happened at several different times, so the running script wereent the same

                             

                            When the hang is occurring, Development would also like to get two separate samples of the fmserverd and fmsased processes about 60 seconds apart. The best way to do this is via the command line, and force the output to be

                             

                            I'll do whenever I have the chance to

                            • 11. Re: Can't add a field on a hosted file anymore, FMS hangs
                              TSGal

                              Vincent_L:

                               

                              The dumplocktable was useful, because it shows a lock conflict on what is presumably the "Main file".

                               

                              Client "iMac1" has a Read lock on the Table Alias catalog.  Client "FMPServer" has a Write lock on the Table Alias catalog and is trying to upgrade the lock to Exclusive, but is blocked by the Read lock.  Client "FMPServer" also has Write locks on Field Definitions and Record Data for a specific table in the same file.  It appears clearly Client "FMPServer" is the customer trying to change the 2 fields in the "Main File".

                               

                              Do you have any idea what Client "iMac1" was doing at the time?  Look at the Access.log file to see more details about this client (IP Address, etc.).

                               

                              There is no sign of activity in the FMSASE process (server-side scripting).

                               

                              The later samples (3 and 4) of fmserverd show the process is trying to quit, but is deadlocked because the worker thread is still blocked trying to upgrade the lock to Exclusive.

                               

                              TSGal

                              FileMaker, Inc.

                              • 12. Re: Can't add a field on a hosted file anymore, FMS hangs
                                Vincent_L

                                Hum, I don't see an Access.log file, where is it ?

                                iMac1 is a mac in a remote (wan far away) location. But bear in mind that it was at night and that remote location is closed. Unintended. So that remote mac was idle.

                                 

                                BUT, I know what was going on, as I experienced this since FMP 14's connection auto-reconnect feature. As I told you, I took care to disconnect all my users before performing my schema editing, but unfortunately, 14 and above clients (at least some of them), even unattended, then try to reconnect to the server.

                                 

                                That's something I noticed in other circumstance, it can be impossible to disconnect remote clients because they reconnect by themselves due to that new 14 feature.

                                I think there's a loophole here, as the admin, asking for client to disconnect should prevent the disconnected clients to reconnect.

                                 

                                So, I purposely disconnected all other clients, did my schema editing, forgetting that the auto-reconnect of remote client may kick in, and it probably kicked in at the time I clicked on commit schema edits.

                                 

                                To me we're seeing the behavior I described and feared : Filemaker server is in an always waiting situation and doing nothing. To me a write lock should have priority on a read lock (and maybe read shouldn't lock at all), especially if editing schema. That kind of conflict should be thought of, and priorities should be decided.

                                 

                                What we're seing here is probably the consequences of the the auto-reconnect feature, that is adding salt to an injury when schema edits committing.

                                 

                                What can be learnt from this is that I could have left filemaker server for years in that state and it would never ever recover. Because it's doing nothing, just wait.

                                 

                                Also, we clearly have a problem with auto-reconnect as the admin disconnect clients order, can be overridden by the auto-reconnection of the clients.

                                 

                                Finally, don't forget that even after disconnecting all clients (except the editing one) once again, or in some other situation killing the server side script client, did not recover the situation. So even if the locking reading client is disconnected, the situation wont resume.

                                • 13. Re: Can't add a field on a hosted file anymore, FMS hangs
                                  TSGal

                                  Vincent_L:

                                   

                                  If you don't have an Access.log, turn it on in the Admin Console -> Database Server -> Logging

                                   

                                  TSGal

                                  FileMaker, Inc.

                                  • 14. Re: Can't add a field on a hosted file anymore, FMS hangs
                                    Vincent_L

                                    Hi TSGal,

                                     

                                    What're the news about this. This can't go on. I just had to disconnect all my users in full production day because of this and the super stupid add a new field in table view that I clicked on by mistake. All scripts are hung, new users can't connect.

                                     

                                    Moreover, this has been 2 nights in a row that I had to go to sleep at 2 and 3 am because I couldn't edit the schema during the day (due to this), and I had to wait important server side script to finish.

                                    Not to mention that completely kills my productivity as both an in house dev and knowledge worker because what I wanted to do on Monday and Tuesday both required the schema changed I had to do the night.

                                    1 2 Previous Next