1 2 3 Previous Next 30 Replies Latest reply on Apr 14, 2017 5:41 AM by wimdecorte

    The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion

    CarstenLevin

      We do know of Virtual Machines which is working really fine with FileMaker Server. But we do also meet VM's and IT departments where the marriage of FileMaker and VM is not one made in heaven.

       

      The issue is wery often performance with as little as from 40-70 users. Where the same solution performed great with many many more users on the dedicated server.

       

      And then the IT departments will ask us, typically after deployment: "Can we use virtual servers" - typically Windows 2008 - for FileMaker, our answer is:

      "If your virtual server is performing as good with FileMaker as the dedicated hardware and software, not only in vitro, but in real life conditions, then the answer is yes!"

       

      This is of course not really a fair answer, or is it?

       

      The point is that the VM experts/IT departments do often claim that their environment is setup to perform even better than dedicated hardware, and then the trouble starts.

      We have heard of many problems with MS SQL, MySQL and Oracle systems in VM environments. And we have heard of and met problems with FIleMaker Servers on VM's. As well as servers running fine for years.

       

      We always try to explain that: "The reference is a well configured fast dedicated server."

      If something is wrong, a bottleneck in the disk i/o, data/processor i/o, network i/o or some other bottleneck, this is not something we can help you with, from a FileMaker Server Perspective. It is the VM that has to perform as the dedicated server.

       

      We do have issues where the IT of large corporations are comming to us when FileMaker is performing awfull in their VM envirionment. We have seen two types of bad performance, but probably with more than two types of causes. While FileMaker is performing bad:

      1. The Processor is nearly not having any load.
        or
      2. The Processor is heavely loaded, more than 100%

       

      In some cases it is badly designed database solutions (FileMaker developers who are self taught and believe that being able to build with FileMaker is the same as being able to build real business solutions for many users. But in many cases it is because setting up VM's is a very complex task. And there are very few real experts.

       

      What do we do then?
      They are always either:

      • Asking: "Why is FileMaker not performing.How should we set up the virtual machine?"
        or
      • Telling: "Our VM is a perfect setup. Why is FileMaker not performing?", Assuming that there is something wrong with FileMaker.

       

      If the solution was previously running fine from a dedicated server, then the issue is most probably problems within the VM environment. Have a look at:

      • Disk I/O
      • Network I/O
      • everything else I/O

      The problem with solving the problem is that we have seen fine FIleMaker solutions that was used by more than 100 users without problems on dedicated servers, but running slow and with long pauses on VM's. While at the same time the Windows OS activity monitors for processor and I/O indicated that the server had no problems.

       

      Now have a look at the FIleMaker Server statistics. Here we will often see that there are long waits for access to data I/O. Very inconsistent with the VM OS measurement!

       

      If we know that the solution is running perfect and fast on a dedicated server, then the conclusion is that whatever the IT/VM experts says, their VM is not performing.

      The Dedicated Server is, in our opinion, the reference.

       

      And then the questions arrives: How should a virtual environment be set up for FileMaker. And since we are not VM experts we will have to answer: "It has to be set up in an way that ensures that FileMaker Server and the users are getting the same performance as they do on the dedicated server".

       

      But I would like to be able to come up with better suggestions or even with a best practice document by FileMaker or somebody else.

       

      What are your experiences, what is your advice?

       

      Best regards


      Carsten

       

      PS. There is a major difference between FileMaker Server pre 12 and 12. 12 is 64 bit while previous versions was 32 bit. This is very important in the virtual environment as well.

       

      Small bibliography with references to the issue - but none really addressing the issue

        • 1. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
          PointInSpace

          We use Parallels Server and VMWare ESXi on late-model Xserves,

          back-ended to 15k RPM SAS, RAID 10, Fibre Channel SANs.  Dedicated RAM

          allocations as well.  We've had nothing but great performance with these

          configurations running FileMaker Server.

           

          There's a huge range of hardware configurations for VMs.  I wouldn't

          be surprised at all if a VM running on sub-par processors, with SATA

          hard drives performs horribly with FileMaker Server.  A VM is not a VM

          is not a VM - and you definitely get what you pay for.

           

               - John

          • 2. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
            aeonSydney

            We've been running FM Server 11 and 12 in VMWare ESXi 4.1 envoriments for some time. The vCPU's is what gets you in trouble. Even though FileMaker server is multi-threaded and supports multiple CPU's it performs better with 1 vCPU. In the WMWare enviroment, you will always get better performans with mutiple users (not the case for single user using it) with 1 vCPU verses 2 or more vCPU no matter how many physical cores or CPUs you have. The reason for this is that when a multi-threaded process asks for another CPU, VMWare has to wait for the other core/cpu to be free before it allows the first CPU to do anything. Essentially the CPU's instead of running independantly, they have to wait and sync. This is why you are getting freezes and the CPU load in Windows Task Manager is low. The (CPU) cost of this process is greater than letting FileMaker serer multithread to another process on the same vCPU. The way ESXi works, all the management of CPUs uses the first CPU. Essentially, a VM that is set for 2 vCPUs is actually using 3 CPUs/Cores. If you ssh into your VMWare console and run esxtop, you can see this happening.

             

            The bad news about this is that there is no way to really use multple CPUs in FileMaker server effectively under VMWare ESXi. (The same is true for MS_SQL and MySQL.) However you can get pretty close to dual CPU performance if HyperThreading is enabled in the BIOS and in VMWare ESXi. Hyperthreading looks at the CPU insturction set and if possible will split it to multiple cores with a CPU. VMware can utililize this and doe so very well with FileMaker Server. So when FileMaker starts a new thread on the CPU, VMware and Hyperthreading can split it up between mutiple cores independent of the Guest OS. However Hyperthreading only works between cores on a CPU and will not thread across physical CPUs.

             

            VMWare ESXi is still a really good way to go for most instrallations of FileMiker server. For a well tuned FileMaker solution, ESXi can handle up to 100 users without any issues. Also if you split up the FileMaker Web Server and FileMaker server into 2 sperate VM, you will get better performance.

             

            -Scott

            • 3. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
              wimdecorte

              Is this specific to ESXi 4.1?  There have been a few VMWare installations where I've seen serious benefit with increasing the number of processors to the instance.  I have never seen any performance decrease.

              • 4. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                aeonSydney

                I have not yet done any testing with 1 vCPU vs 2 (or 4) vCPU with Filemaker Server on ESXi 5.x. From what I have read, the same problem still exists with MS SQL and MySQL, however there is a performance boost of about 8% on average with 5.x

                 

                Keep in mind that if there is no load on the server, on a dual 4 core cpu server (8 cores total), when I did the testing, a 2 vCPU, 4 vCPU both performed faster for a single user load. However, when you have multiple users on and some load on it, it freezes and crawls with esxtop showing high CPU and Windows Task Manager on the FileMaker server showing low CPU usage. Here is my benchmark results running a complex report with lots or summarries and sorts. (Note that the clients and server was restarted between each test to ensure there was no caching envolved.)

                 

                # Simultaneous Users     v CPU     Time (Each User)

                             1                          1                    18 sec

                              3                          1                    22 sec

                              1                          2                    11 sec

                              3                          2           2 min 15 sec

                              1                          4                    10 sec

                              3                          4           4 min 30 sec

                 

                A lot of improvements were made with 5.x to take advantage of HyperThreading from what I undersand. I plan on doing the same benchmark with 5.x somtime soon.

                 

                -Scott

                • 5. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                  user24968

                  Sorry to reopen an old post, but we seem to be seeing this on FileMaker Server 13 & 14 on VMWare ESXi (not sure what version, but I think it's 5.x - I don't control the VMs).

                   

                  Running benchmarks (Neo Code Speed Test) we see a nearly 2x increase in speed when we drop from 4 vCPU cores (or indeed 4vCPU sockets) to 1. This is mirrored in real world performance with some routines in our solutions.

                   

                  We've got an open ticket with FMI about this, but does the community have any feedback as to how we might increase the cores (so we can scale the solution)? Or can we stick with 1 core for 30-50 FMP users if we hive off the Webdirect (around 30 users) to a separate server?

                   

                  Cheers

                  John

                  • 6. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                    0614775139

                    We have just started experiencing issues as initially described with poor performance on VMWare 5.02?? not quite sure as I don't control the servers either.  FM  Server 13.0v10 Windows Server 2008 R2 Enterprise SP1

                     

                    I haven't tried reducing the cpus will see if I can get it dropped back to 1 to see what happens.

                     

                    John, (user24968) if you get a response from FileMaker or solve it please post back in this thread what you did. I will do likewise.

                     

                    Cheers,

                     

                    Chris.

                    • 7. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                      CarstenLevin

                      We are having a very very fast and well configured (dedicated) server running with one of our large installations. And FileMaker Server (14.0v4) is definitely utilising all the processors. This very large system is in heavy use by up to +200 users and is performing splendid.

                       

                      So it is not so much a matter of FileMaker not utilising more than one processor ... when you use a VM it is a matter of whether this VM is giving access to those resources in an appropriate way ... or not!

                       

                      FileMaker has changed la lot over the latest 3 iterations. Please try testing on the very well configured dedicated server first and then kindly ask your VM specialists to match this performance.

                       

                      Best regards


                      Carsten

                      • 8. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                        0614775139

                        Thanks for your reply Carsten,

                         

                        So it is not so much a matter of FileMaker not utilising more than one processor ... when you use a VM it is a matter of whether this VM is giving access to those resources in an appropriate way ... or not!

                        I appreciate that but if I go back to my client's IT department and tell the to configure their VM "appropriately", well I think you can guess what the response will be.

                         

                        FileMaker has changed la lot over the latest 3 iterations. Please try testing on the very well configured dedicated server first and then kindly ask your VM specialists to match this performance.

                         

                        The whole site has gone Virtual. The only dedicated server there is the Win 2003 server running the previous FM11 version of this solution.  I have asked them to install Win 2008 server on it so I can test it but they are hesitant as it is a 5 year old server and they see no reason why FM shouldn't run properly on VMware.  They have no issues with any of their other applications and we are talking hundreds of them.

                         

                        Some numbers, this is the stats when it was running with 6 cores, I/O and is low but wait and elapsed are huge.

                        6 Cores.jpg

                        The in progress calls sit here at a value of 1 for minutes at times, when this happens the workstation is frozen.

                         

                        I had the whole team looking at it yesterday, guys monitoring switches, disk and network i/o on the VM etc. Everything running at under 25% (mostly lower). CPU cores were averaging 10% and spiking to 40% very occasionally.  Everything pointing to FM Server being the bottleneck. Why the long Wait time??

                         

                        I got them to drop it to one core and it is definitely better until the CPU hits 100% which is expected. The same report loads in about one third the time and there are no long 2 or 3 minute long freezes on random workstations which are waiting for the server. 1 Core see below:

                        1 Core.jpg

                         

                        The wait time has dropped significantly.  Elapsed time is still high but the cpu did hit 100%. Unfortunately 1 core is not a viable solution as with 1 core the CPU hits 100% a lot. So the question remains, why would increasing the cores reduce performance of FM but improve performance of all the other apps they are running on the same box?

                         

                        It may be possible to to tweak a setting somewhere or tick an option that will fix this for FM but I don't know where to point them to look and from their point of view it's an FM problem...........

                         

                        Regards,


                        Chris.

                        • 9. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                          jeffbeck123

                          We have been running into the same issues as well.

                           

                          We are running FMS 11 on 2k8 R2. I have yet to try the 1 cpu trick but I will attempting to do that tomorrow as well.

                           

                          Chris next time you talk to your VM guys ask them to increase the CPU shares allocated to your FM box. This should give your VM a bigger piece of the CPU pie regardless of how many CPU's the box actually sees.

                           

                          For example if you have 10000 CPU shares to go around and 5000 is dedicated to your FM box that means ever 2nd cpu request will be dedicated to your box even though it only has 1 cpu and the other machines running on the same VM server have 10.

                          • 10. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                            CarstenLevin

                            Hi Chris,

                             

                            Yes I understand you very well: But you need to cut the issue in two parts:

                            Do you have a problem with your solution. Is the architecture problematic and does it need optimising?

                            Does your user interface and the depth of related data showing or depth of unstored calculation drag down the server?

                            Etc.

                             

                            This you should test with a dedicated server.

                             

                            When you know if you are having a problem or if the VM people are having a problem you should point them to the dedicated server.

                             

                            Unless you are a dedicated VM and Windows server specialist you should and can not solve their problems, they have to accept that your solution is working very well on Windows Server ... if it is ... and then they have to make their server work as well or give up and give you  a dedicated server.

                             

                            Btw: It is possible to make FileMaker run very well on VM servers, but only if the people behind the VM's are very competent.

                             

                            And just one more question: Is it not strange that they are not using Windows Server 2012R2?

                             

                            Best regards

                             

                            Carsten

                            • 11. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                              tomdc

                              Hi,

                               

                              I'm having a similar problem here at a university site.

                               

                              After moving a rather heavily used application from a 6 year old (dedicated) XServe to a new Virtual Machine (vSphere 5.0; Windows Server 2012R2; originally 2 vCPU's and 8 GB RAM), there was immediately feedback from the users that the application ran a lot slower, with frequent periods of "hangups", followed by periods were it seemed to be better for some time.

                               

                              FM stats showed significantly raised "Elapsed time/call" and "Wait time/call" figures (up to 600-800k, or even higher), where "Disk KB/sec", "Network KB" and "I/O time/call" all remained low.

                              When looking at the Windows Task Manager, I noticed that during the periods of high Elapsed & Wait Times/call for FMS, the CPU % utilization seemed to be "capped" at about 50%, where the CPU graph showed some kind of "plateau".

                              As soon as the CPU % utilization dropped again to lower levels, the FMS Elapsed & Waiting times dropped immediately as well, whereas as soon as you could see the CPU go back to its "plateau" the FMS elapsed & waiting times went up again as well.

                               

                              The exact same behaviour happens with FMServer 14v4 as well as FMServer 13v10.

                               

                              Although the central IT-department, which maintains the virtual servers, claimed there was no need to raise the CPU count (as the cpu utilization never came above 50%), they agreed to upgrade the VM to 4 cpu's and 16 GB of RAM.

                              This "upgrade" had no noticable effect on neither the user experience or the FM stats; but in the Windows Task Manager all 4 vCPU's seemed now to be capped at about 25% utilization now whenever the Elapsed & Wait Times went up!

                               

                              This looked to me as being some kind of configuration error of the VM (the VM seems te be limited to 1 "effective" vCPU, wheter it is 2 vCPU's @ 50% or 4 vCPU's @ 25%), but the central IT department assured me it's configuration was ok (it's a university IT-department, so I'm pretty sure they now a lot more about VM's and Windows Servers than I do).

                               

                              Currently the application is running from the old Apple XServe again, without user complaints & "normal" elapsed & waiting times.

                               

                              Any ideas, or has anybody been able to solve this "VM issue" by changing the VM configuration?

                               

                              thanks for any help,

                              Tom

                              • 12. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                                wimdecorte

                                tomdc wrote:

                                 

                                Although the central IT-department, which maintains the virtual servers, claimed there was no need to raise the CPU count (as the cpu utilization never came above 50%),

                                 

                                That is a typical discussion but it is an entirely useless one.  The only stats that count are the FMS ones, not the OS ones.  If FMS shows elevated elapsed & wait times per call it is because it simply does not have horsepower in the machine to honor the client requests.  And there is only one solution: more horsepower.

                                 

                                With one big caveat: sometimes the design of the solution is such that increasing the # of cores available doesn't really help.

                                 

                                The issue really is only sideways related to the VM aspect: you would see the same result if you put the solution on an underpowered physical box too.  IT will always try to create VM instances as lean as they can and obviously the more instances the physical  host machine has to handle the scarcer the resources are for all of its VMs.

                                 

                                The long and short of it: let the FMS stats be the guide and bump up the horsepower until they are all in the green.

                                • 13. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                                  tomdc

                                  Hi Wim,

                                   

                                  thanks for your fast reply!

                                   

                                  The IT-department did raise the number of CPU's dedicated to this VM to 4 (instead of 2), but this had no real effect at all... What surprised me, was that with 4 vCPU's, the fmserver process never used more than about 25% of the "Total CPU usage" - it stayed hoovering between 20 and 25% for minutes, while FMS struggled (elapsed & waiting times through the roof).

                                  Before that, when the VM had 2 vCPU's, the fmserver process did the same, but at around 50% of "Total CPU usage".

                                   

                                  The fact of going suddenly rom 2 x 50% to 4 x 25% makes me think it either is a VM configuration issue, or an issue between FMS and the VM software, which makes that FMS for one reason or another does not use all available CPU power.

                                   

                                  I think the design of a solution can always be improved, but as the same solution performs a lot better on an old Xserve (2,26 GHz Quad-core Xeon ; 12 GB RTAM; from 2010) I don't think that's the biggest problem.

                                  • 14. Re: The VM issue - FileMaker Server 10-11 and indeed 12 - Discussion
                                    wimdecorte

                                    tomdc wrote:

                                     

                                    The fact of going suddenly rom 2 x 50% to 4 x 25% makes me think it either is a VM configuration issue, or an issue between FMS and the VM software, which makes that FMS for one reason or another does not use all available CPU power.

                                     

                                     

                                    Not an issue with VM config or FMS and VM; you would see the exact same thing if you had a matching physical box.

                                     

                                    FMS is fully multi-threaded but only to some extent (based on design).  There are lots of scenarios where FMS would not be able to use all of the CPU power.  The most common one is if you have one table that pretty much all clients need to write to in most of their interactions.  If you do then it doesn't matter that the machine has 1 core or 100; the index on that one table has to be updated sequentially and every call will have to wait his turn.

                                    Another frequent issue that will make FMS "stall" but not use all of the available processing power is letting people do searches on unstored calculations.

                                     

                                    The extra cores can and will be used by things like server-side schedules and PSoS sessions since those are separate session outside of the normal FMS processes.

                                     

                                    The old X-serve has 4 cores but those may support HyperThreading which really means it has 8-cores.  That would still be double than the VM has after IT upped the core count.

                                    1 2 3 Previous Next