11 Replies Latest reply on Jun 12, 2017 11:44 AM by CICT

    How to continue development on active product?

    marksystech

      Hello all,

       

      I'd tried to find a search for this and really could not locate what I was looking for.

       

      I've completed the first "module" of a FileMaker project for a client.  This is a fairly elaborate purchase order system to be used by the employees of the company.  It is deployed on a FileMaker server.

       

      The company is interested in moving into the next phase to develop a materials tracking system that they want to be part of the same "portal" as we call it.

       

      This is where I'm not exactly sure how to proceed.  They will be actively using the FileMaker product and "module" I've created.  I can see them requesting bug fixes and now the new development.

       

      For the bug fixes I had planned to remote in and fix "inline" on their actual database after testing on on a local copy.  This will be ok for bugs and minor updates.

       

      But how will I go about adding the new modules to their existing database?  I may have simply missed it but is there a way to "import" forms and tables when I'm ready to?

       

      I do remember reading something about how to do this quite some time ago but I cannot seem to find that reference.

       

      Thanks in advance

        • 1. Re: How to continue development on active product?
          planteg

          marksystech ,

           

          a 'portal' is part of FileMaker, i suggest you use a different name to tell us what is a portal for you customer if we need to understand what it is.

           

          Unfortunately with FileMaker it's not as easy as with other tools to implement modular development. With FileMaker Pro Advanced, you can copy some objects from one file to another, but not  all of them.

           

          Todd Geist has work on the concept and has put together a method you can see by visiting http://www.modularfilemaker.org/ This method is a few years old, but you may find the method interesting.

           

          Next week someone in Montréal will present a method for modular development. I will not attend but work with someone who will. Hopefully he will learn a few tricks.

          • 2. Re: How to continue development on active product?
            philmodjunk

            Data Separation is a method that puts your tables in one file with layouts, scripts, etc. placed in a second file. This allows you to replace an older copy of the interface file with a newer copy without the need to import data.

             

            Changes to the data file still require importing records from the old to the new if you choose to replace files, but table definitions and field definitions can be copy/pasted so you can transfer changes to those elements from one file to another.

             

            Any changes made to the live file, however small, should be made with careful forethought and user coordination. Any change that you make could catch them in the middle of a task with unforseen consequences.

             

            Changes made via Manage Database are particularly risky. It's quite easy to lock all users out of editing data in an entire table. Haven't tried this since FMP 10, but back then, simply opening the calculation editor for an auto-entered calculation locked the table until I closed the dialog box even if I made no changes. There's also a small chance that a "glitch" could damage your file if it happens while a change is being committed to the file.

             

            Last time we discussed this here, others disagreed, but I never take that particular chance. We build a "to do list" of data level changes in a table in the solution and take the files down off the server after hours one day a week to make those changes.

             

            There's also a tool called FM Refresh that you might want to take a look at.

            • 3. Re: How to continue development on active product?
              marksystech

              Thanks Planteg,

               

              Yes I know portal is a FM component.  This company calls this project I made for them their "Access portal".  I'll try hard not to confuse the two here on the forums.

               

              I'll check out the modular stuff.

              • 4. Re: How to continue development on active product?
                marksystech

                Thanks Phil,

                 

                I'll check out refresh.

                 

                My biggest concern is the addition of new code/layouts/tables.  They will be using the product and at some point after I've got things working on the next phase I'll need to as you say take things down and manually make changes which could be very tedious.  Hoping to find a better way but there may not be one.

                • 5. Re: How to continue development on active product?
                  wimdecorte

                  In essence you only have two choices:

                   

                  1) redo all your schema development changes in the live system - this saves you from having to transfer the data

                  2) make your dev copy the new live copy, and transfer all the data

                   

                  The data separation model can alleviate the pain here by allowing you to swap the UI/scripting file and not have to worry about transferring the data or redoing schema changes.

                  In a system under active development where tables and fields are added and changed, the benefit is a little less but still saves time and effort and risk.

                   

                  Both core choices carry risk.  That risk in both scenarios can be mitigates a bit by using tools.  In scenario #1 by keeping good notes and making use of diffing tools.  In scenario #2 by using a tool like RefreshFM to automate a lot of the data transfer.

                   

                  You'll find that you end up using both approaches depending on the nature of the deployment.  One approach is not inherently better, faster or less risk than the other.  It all depends on the quantity and nature of the changes that need to be implemented.

                  • 6. Re: How to continue development on active product?
                    philmodjunk

                    Hoping to find a better way but there may not be one.

                     

                    I wonder if you really read my entire post. What I was describing was a "better way". In particular, data separation allows you to modify a copy of an interface file and when your review and testing process is complete, you just remove the current interface file and replace it with the new.

                     

                    Note that layouts, scripts, field definitions and tables can all be copy/pasted from one file to another. (Technically, you can't copy/paste a layout, but you can copy/paste all objects on the layout in one copy/paste action and that does have significant potential issues that you have to consider before doing this, but as a way to update a production layout that you've "tweaked" in a development copy of the layout, it's often an easy thing to do.)

                     

                    We maintain a 100+ file system with over 300 users by taking it down for updates for what is usually an hour or less each week. Granted, with a new system, you'd need time to do more, but we are only doing a small portion of the changes while the system is off line. The rest of the updates we do to the live system, but we look before we leap and make lots of back ups on a regular basis as well as making a lot of changes in ways that allow us to easily and quickly revert a change in the unlikely event that something we did didn't work out as we expected.

                    • 7. Re: How to continue development on active product?
                      beverly

                      Ditto! If you have something that works and need to add new "modules", separation can be your friend here. Keep what you have, but create a new "interface" for the new module (new database, new file). You may choose to put tables of data into this file or put them into the existing database/file(s). Either way your new interface can point to the current table(s) as needed via the relationship graph. You can navigate between the "old" and the "new" without making any changes in the old (unless you add data tables there).

                       

                      beverly

                      • 8. Re: How to continue development on active product?
                        og1

                        Hi all.

                         

                        I've been researching this exact topic for multiple weeks now.  I see no "formal" Filemaker tool or methodology to pull from the production database, make changes to the files or files in a standard development environment (while the production version branch continues), then merge the changes from both the development and production versions back to together within the production branch. 

                         

                        If I was working an a traditional relational database and web based user interface I'd use standard tools such as GIT and a continuous integration tool to manage this process.

                         

                        I'm very surprised that Filemaker touts it's product as an integrated development and operations platform, but this standard functionality that blends a development version with the ongoing production version changes is not a core component/function of Filemaker.

                         

                        To date I've been able to get away with this manual process to develop changes to the existing files and then put the changes back into the production version.

                         

                        1) Copy production fmp12 files to development environment (work continues on production version)

                         

                        2) Do what is needed on development version.  Test development version with users (use dummy data on new features where needed). 

                         

                        3) When all is good on development version, manually copy changes made to production version since the last copy/pull to the development environment (I know a script can be done to help with this, but in many cases scripts are not practical or even possible if the changes are significant...can only help so much). 

                         

                        4) When the development and production copies are both in sync (either through manual or automated methods), schedule a maintenance window and replace the production files with the development files.

                         

                        Just not a modern way of doing ongoing systems design and development in 2017.   And if the changes are extensive, it's a major task that has significant operational risks.

                         

                        Someone mentioned the Filemaker standby server Standby Server - Overview | FileMaker as a way to help with the design and management of getting the production data changes incorporated with the development version, but I'm still not seeing how this is going to help.

                         

                        I've done some preliminary investigation into that external software tool called RefreshFM ( RefreshFM | Automate your Updates ), but I'm not anxious to pay for another expensive piece of software after paying so much already for Filemaker vs the open source databases and web development tools that are tried and true and I could be using.

                         

                        Does someone have a proven Filemaker based solution to this problem?

                         

                        Is there someone from Filemaker that contributes to these forums? I've seen no suggestion from anyone from Filemaker to this problem?

                         

                        Thanks, all.

                        • 9. Re: How to continue development on active product?
                          wimdecorte

                          og1 wrote:

                           

                          Does someone have a proven Filemaker based solution to this problem?

                           

                          Is there someone from Filemaker that contributes to these forums? I've seen no suggestion from anyone from Filemaker to this problem?

                           

                           

                          Don't expect any suggestions from FileMaker.  It's a solved problem.  The suggestions in this thread and many other threads clearly outline the options that are available.

                           

                          The solutions are not as graceful as what you get from Git for sure, but they are not unknown either.

                           

                          The lack of proper versioning is a long-standing issue for most of us who are used to some of the versioning tools available in other environments.

                           

                          The operational risks you mention are real.  But known and well documented.  So following a good procedure will mitigate those.  A well established deployment plan takes the stress out of those.

                           

                          On big projects; the time we spend on deployments/go-lives is quantified and included in our estimates and while it is always just a fraction of the total cost, it can be significant; and it can require a fairly long maintenance window and careful planning.

                           

                          It has never been a deciding factor in choosing the platform for a project or not choosing it though, largely because there is nothing unknown about this process.  We've done it thousands of times.

                           

                          Do we wish it would be a lot simpler and more automated: heck yes.

                          If you have not voted this one up yet, please do so:

                          https://community.filemaker.com/ideas/1058#comment-9672

                           

                          Or add your own product idea so that people can vote it up.  FMI does pay attention to these.

                          2 of 2 people found this helpful
                          • 10. Re: How to continue development on active product?
                            og1

                            Thanks for the reply.

                             

                            Question, I do research across many industries for a living and can usually find most the info I'm looking for.

                             

                            But for some reason with Filemaker, I can't find where detailed procedures exist for for enterprise level Filemaker version management and control across development and production environments is "well documented".  I just see that everyone has they own "general method".

                             

                            If someone has a link(s) to the detailed procedure please take some of their valuable time to share the links to the procedures on this thread?  Greatly appreciated.  Thanks.

                            • 11. Re: How to continue development on active product?
                              CICT

                              Hi Mark

                               

                              As I'm doing exactly this at the moment, working outside of normal working hours to add tables and fields, I'll reproduce our procedure.

                               

                              We have a very mature healthcare system used in the UK, Dubai and Hong Kong. As Dubai work Sunday to Thursday and are 3 hours ahead of us in the UK, finding slots to make changes is not easy.

                               

                              Starting from the live system, tonight we need new tables and fields to be added to existing tables. We have a single UI file that does include a few tables for global fields and customised reports that are built dynamically for export to Excel, but no data exists within this file.

                               

                              We have 5 data files that feed into the system for standard FileMaker data, 2 types of document storage, email attachments and audit logs. There are very few calculation fields in the data files and even fewer cross table links.

                               

                              All working and background layouts, scripts, and table occurrences are built up within the single interface file. Each file has to have matching security access privileges entered and these are controlled via a Human Resources system, so that adding and removing users occurs across all files. Calculations and data set automatically are mostly via script triggers and associated scripts, keeping the business logic within the UI file.

                               

                              Tonight I'm adding the new tables and fields in the live data files. The UI file has a version number displayed on the home screen and there is a dedicated versioning layout - some files we've built have a table for us to enter records in, but equally we use a blank layout where we add the versioning information as text on this layout. The key thing is that we know which is the current and which is the development (or even additional test version) and what the differences are. As an extra precaution we have large text on the home page, only visible by the developers stating whether this is a live or development version alongside the server name. We make this visible to all should we ask a client to test the development system before we carry out an upgrade.

                               

                              After the data files have been updated, we will take backuped up copies to our development server. In this case, we have another version of the UI file that has already had some development on it, therefore the version is at a higher level than the live system. We're using Windows servers, so just need to download the revised data files to the development server, close the development files in FileMaker Server admin console, replace the data files, but leave the UI file in place and then open up everything using the admin console again. If we were using Macs we'd need to deal with permissions issues, but that is pretty standard.

                               

                              Now we can continue development. Once we're ready to perform the update on the live system, we take a copy of a backup of the development system UI file and transfer it to the live server. Outside of working hours we close the live files in admin console. We have a procedure working from right to left where we have 3 folders open: on the far right the new UI file, in the middle the current live UI file and on the left an archive folder named with data and time. We move the live file from the centre folder to the left archive folder, we move the new UI file to the left to the middle live folder.

                               

                              It is important to note that we never use the FileMaker Server remove file facility. We currently store about 70Gbs of files in container fields stored externally, these would all be removed if we used this and it is something FileMaker could do to improve their system. Separating the backup of external container folders from a server level to a schedule level is another.

                               

                              Opening the files in admin console upgrades the live system with the new features. Although hardly ever used, having archived off the previous UI file, we are able to downgrade if needed by replalcing the new UI file with the original. Downtime is usually about 2 minutes, so sometimes we can agree an update during working hours. Particularly helpful if we wish to do an upgrade but don't want the Asian or Middle Eastern users to be the first to the system, allowing the UK users to test first.

                               

                              Any new users added to the live system since the development version was taken have to be added (they already exist in the live data files). These can be added manually in the UI file and our Human Resources system allows us to easily reset their passwords - these users are warned in advanced. Any changes to passwords during this time also have to be accommodated similarly.

                               

                              Normally we'd leave the new UI system in place for a little while to allow all the passwords to be reset before taking another development copy. Versioning will have been updated to record when the update took place, who carried this out and there will be an overview of changes that have taken place.

                               

                              I think this covers most of the procedure and hope it is of interest.

                               

                              Kind regards

                               

                              Andy

                              1 of 1 people found this helpful