7 Replies Latest reply on Dec 29, 2014 3:37 PM by wimdecorte

    Mbps- Containers over the WAN

    smith7180

      Hi,

       

      I'm a neophyte when it comes to all things WAN.  I don't want to waste anyone's time having to explain WAN basics to me, but I would like to pose a general question to preview/guide my research over the next few days (I've also scheduled time with a professional).

       

      My situation- I've created a database for the media sector.  It has lots of images  ~1.5MB each (1500x1800 pixels usually).  Generally speaking I'm either displaying one medium (~400pixels square) per page, or lots (10-200) of smaller (around 45 pixels square) images in a portal.  Everything's super fast on the LAN.  Instantaneous really.  I've devoured every post/blog/article I could find on performance and tried to apply the tips in my solution.  No unstored calcs, merge fields in list view on ios, relationships instead of sorted or filtered portals, blank layouts where helpful, no flushed caches, only native filemaker objects on layouts, data across multiple small tables instead of fewer large tables, no new windows and very minimal navigation to layouts in my scripts, etc...

       

      I feel like it's paid off.  The database is snappy on OSX, Windows 8, iPhone, and iPad.  That's probably par for the course of the average TechNet visitor, and the forums have been a great boon for me.

       

      Then I put it on a Filemaker server 500 miles away.

       

      Actually when no images, or even 1 medium image (400 pixels square vs originals around 1500 pixels square) are displaying- it's better than I had anticipated.  When I get those portals with images though- big slow downs.  These are the 45x45 pixel thumbnails.  I enabled permanent storage for thumbnails in my database (though perhaps some reciprocal action need be taken on the server?), but it hasn't seemed to make a difference.

       

      Here's the thing that has me confused but hopeful- when I monitor network traffic, it basically hovers at only 1.2mbps.  This has been the case on 2 different hosting services (both of which I was happy with).  I've got around 100mbps, and the host has significantly greater bandwidth.  Leaving aside any mistakes or bad practices I've committed, is it within the realm of possibility that I could get filemaker server to be sending me pre-generated (ie permanently stored) 45x45 thumbnails at a much greater rate (say, >20mbps)?  We're talking about scrolling a portal where everything else in the layout is already loaded and no scripts are running.

       

      Thanks!

       

      Edit: I should have mentioned that my containers are stored in secure external storage on an ssd.  I've tested a self-contained copy as well with the same results (ie ~  1.5 mbps)

        • 1. Re: Mbps- Containers over the WAN
          monkeybreadsoftware

          I'd highly recommend to calculate on the servers thumbnails. In your case 400px, 45px and other sizes you need. Put them in different containers. For example in the table has Container for Original, Thumbnail400, Thumbnail45 and other sizes.

          In the layouts, always pick the field in the right size.

          If user modifies such a container, be sure to move the new image into the original and update the thumbnails.

           

          You should not move dozens of 1 MB big pictures over internet to client to show them scaled down as thumbnails.

           

          Greetings

          Christian

          • 2. Re: Mbps- Containers over the WAN
            Mike_Mitchell

            Hello, Smith.

             

            Let's review the behavior you're seeing:

             

            1) Quick on LAN.

            2) Quick with no images or only a single image.

            3) Crawls when displaying a portal.

             

            So the two differences are:

             

            1) WAN

            2) Portal

             

            First, bandwidth isn't really your problem (which you already know). But latency can kill you. If you add, say, 50 ms of latency to your connection, then each packet that travels over the network will take that much more time. Aggravating the situation is the way FileMaker loads data from the server.

             

            When a client requests data from the server, it receives it in blocks of records. The number of records will depend on the view being used. For Form view, the data are sent in chunks of 25 records. For List view and Table view, the records are sent in chunks large enough to be displayed on the user's screen (i.e., however many records will fit).

             

            Portals are unique. In the case of a portal, all related records are loaded when the parent record is requested (with some exceptions). Uh oh.

             

            That leads us to behavior #2: When records are loaded, all fields in the table are loaded (with a few notable exceptions), even if they're not displayed on the current layout. So if you have a table of, say, 5 fields, it's going to load substantially faster (rule of thumb) than a table having 50 fields. This is, of course, dependent on the amount of data present in each field.

             

            So how do we fix it? We have to compensate for the dramatically increased latency on the WAN, and to do that, we have to reduce the amount of data being pushed over the wire. This gives you some strategies to pursue:

             

            1) Narrow your tables. Get rid of any fields you're not actively using, or consider using a parallel table and a 1-to-1 relationship to show just the information you need in your portal. And speaking of portals ...

             

            2) Get rid of the portal, if you can. A List view will load periodically, rather than all at once, leaving your user will little "stutters" in performance, but it will break up the loading so he won't be sitting there twiddling his thumbs waiting for the thing to load. (Users hate that.)

             

            3) Reduce the complexity of the layout. Use the provided themes and don't layer objects. (You may already be doing this.)

             

            4) If you do decide to split out your table using a 1-to-1 relationship, consider which fields are frequently updated versus those that are only updated once in a while. The ones that receive frequent updates will have to be pushed down to the clients to refresh their local caches, so they should be together, leaving the infrequently fields separated out. This will reduce traffic during idle times.

             

            5) Another option is to use managed (external) containers. They will typically process faster than embedded containers. (You didn't say whether you're using this feature or not.)

             

            One thing you can do is, if you have access to the Admin Console (might not, since it's a hosting service), you can check the Remote Calls / sec, the I/O Time / call and the Wait Time / call statistics on the server. This will give you an idea of what's going on at the server end.

             

            Those are just some general principles. If you give us a little more detail on how you have this set up, we might can provide more detailed or specific advice.

             

            HTH

             

            Mike

            1 of 1 people found this helpful
            • 3. Re: Mbps- Containers over the WAN
              wimdecorte

              smith7180 wrote:

               

               

              Edit: I should have mentioned that my containers are stored in secure external storage on an ssd.  I've tested a self-contained copy as well with the same results (ie ~  1.5 mbps)

               

              Remote container storage is faster than embedded old-style.  In one of the previous devcons a FMI engineer told us that old-style containers are retrieved in 4k increments, remote container data is retrieved in larger chunks.

               

              So you should see some impact, but as Mike already mentioned, latency may be bad enough that it negates the bit of pure performance benefit you get from using remote container storage.

              • 4. Re: Mbps- Containers over the WAN
                smith7180

                Mike-

                 

                Huge thanks for the detailed reply.  I realized not long after I posted my message that I forgot to mention I was using external storage (secure) and added in an edit.  In the planning weeks I will be building my first server so I will certainly look forward to the Admin console's stats.  I may rent a dedicated server in the meantime to get that info quicker.

                 

                One thing I might added is that it is the stuttering I get while dragging this portal in FMPro is totally absent in web direct and iOS.  In both those cases everything remains quick and smooth- the portal scroll bar does not stutter.  However the images will continue take longer than I would expect 10 45x45 jpgs to load over a swift connection.  I guess in FMP adv the portal waits till the images are received before it will scroll?   Nevertheless it was a bit of a shock to find my database more responsive in webdirect then FMP advanced.

                 

                I have indeed tried to narrow my tables.  The images form their own table, so I'm hoping that when I load one inventory record (which display the single 400x400 image from the related Images table) in form view, that the 25 other records don't include their images.  Additionally all 'non-basic' facts about the inventory item are in related tables in an effort to keep it narrow (a table for notes, tags, transaction history).

                 

                I confess I could do with less complexity in the layout.  I have a weakness for conditional formatting .

                 

                Not knowing enough about networking (yet), I know I have to abstain from any judgment or assumptions.  However based on what I was seeing (a fully loaded layout pausing to load 45x45 images, it was/is so tempting to think that either the server was not saving the thumbnails and/or something was crippling FMS's ability to spit out the data.

                 

                I will need to research all things latency.  Again, I very much appreciate your detailed response

                -J

                • 5. Re: Mbps- Containers over the WAN
                  smith7180

                  Thanks Christian!

                   

                  I guess I had assumed that filemaker server was doing precisely what you suggest I do manually:

                  Managing performance with thumbnails

                  To speed up the rendering of images in container fields, FileMaker Pro by default generates image thumbnails and caches them in memory....

                  Permanent storage caches on-disk in addition to in-memory. The on-disk portion of the cache remains when the database file is closed.

                  Either way, what you describe is definitely what I want to happen.  Perhaps I should look into doing it manually as whenever the image is displayed in the medium or small format it is not editable so I don't need to worry about moving new images.

                  • 6. Re: Mbps- Containers over the WAN
                    smith7180

                    Thanks Wim,

                     

                    I confess I don't have a strong grasp on the issue of latency and the impact it can have.  I look forward to researching all these issues in the coming weeks.  Nevertheless, it's hard to look at a fully loaded layout with a simple portal of 45x45 images in a database with permanent thumbnail storage and wonder why filemaker server is only sending 1.5 mbps.  I'm sitting there- no network traffic, scroll down in a portal with no other special formatting or tricks, and I never go above 1.2 mbps.  On two entirely different hosting services.  Could that be latency?  My understanding of that particular issue is goes no further than pinging a server (and though I didn't ping the filemaker servers, pinging public servers in the vicinity gives me 32ms (no doubt I'm missing the point though.  Hopefully whatever I learn will eventually result in better performance).

                     

                    Thanks again-

                    J

                    • 7. Re: Mbps- Containers over the WAN
                      wimdecorte

                      smith7180 wrote:

                      and wonder why filemaker server is only sending 1.5 mbps...

                       

                      No idea how you are measuring that but it would take a complex set of metrics to figure out what bandwidth FMS is actually using.  It is only going to be as fast as the slowest connection on the path between FMS and you.  And that's a big set of hops.

                       

                      Also note that bandwidth is just how wide the pipe is.  Latency tells you how fast data is going through that pipe, and is more important in this case.