14 Replies Latest reply on Jan 10, 2012 2:04 PM by CarstenLevin

    Advice on "where to now" - seperation vs single

    kiwikaty

      Hi there

       

      I am an inhouse developer for a solution that is 10 years old, Originally in Filemaker v5 and was 102 files, with the release of v7 it was converted down to 13 files, and now more recently I have created a UI file for the interface but the "data" files still contain the layouts and scripts that formed the basis of the UI (incase I had to backtrack).

       

      The users have been in the UI file for 12 months now.

       

      I am trying to make the decision whether to now make the UI file 1 big data AND UI file which would effectively consolidate the following files:

       


      FileSize
      Students
      2.1GB
      Coredata1.9GB
      Practicum295mb
      Programmes91.8mb
      Classes
      78.8mb
      Codes
      44.7mb
      UI
      32.2mb
      Staff
      14.2mb
      Main Menu 1.28mb


      Which in one folder = 4.55GB

       

      The UI currently has just over 1000 TO's which would all need repointing to local tables once I imported them and the broken calcs would need uncommenting... (this could cause some RSI as about 3,700 of the 10,500 fields are calcs!).

      Some futher TO's would need to be to be added that are not yet created in the UI as I did not need them for scripts or portals but will need them for calcs if I make this change.

      Then the data would need importing. Not sure that it would be possible to get this much data imported over a weekend period?

      I think it would be about 2 weeks solid development work (not sure if this is optimistic or not) and I would have to put a freeze on any schema changes in the live system during that time.

       

      OR

       

      The other option in to leave the data files separate but go though them and clean out all the layouts and scripts so they really are more like data files. This would be less work and potentially has a lot less room for errors but the TO's in these files are a REAL MESS whereas the UI file has been kept fairly clean in terms of anchor bouy. Truth be told I am always a bit embassaed when I look at the graphs in these files!

       

      There are people that do not like the separation model but then there are others that swear by it. I am not fussed either way but I want to make a sensible decision and then get on and clean things up either way. As our solution is not a shrink wrap we do not have a need to "swap out" the UI, its creation was originally motivated by a corrupted file that needed re-doing from scratch and it grew from there once I realised I could develop it alongside the production version without having to re-import the production data.

       

      We are lucky enough to have a Kove server so backup time of a large file would not be an issue.

       

      I have always thought it was best to have the data separated out into multiple files in case we have a file corruption and need to reimport the data (as this can be very slow… days slow) but there seem to be two schools of thought on this also?

       

      I am not scared of a hard slog but definately want to make sure the path I go down is the better advised one before I start out. I hope you do not mind me using the forum to seek some opinions.

       

      Kind regards

      Katy

        • 1. Re: Advice on "where to now" - seperation vs single
          ibrahim_bittar

          Hi Katy

           

          I'm not (anymore) a big fan of data separation but in your case I think it's the best way to go.

           

          Try to add some order in the relationship graph: use colors, notes and groups, so you can see the whole thing without making your eyes cry :).

           

          Saludos

           

          Ibrahim Bittar Torres

          Director General

          Eikonsys, S.A. de C.V.

          FileMaker 10 Certified Developer

          http://www.eikonsys.com

          FileMaker Business Alliance

          • 2. Re: Advice on "where to now" - seperation vs single
            PSI

            Katy,

             

             

             

            Flexibility of backup is good reason to leave the data files, especially the larger, ones separate. I think your biggest bang for the buck is the ‘or’ option…clean up the data files. Deleting the layouts and scripts might help but cleaning up the relationship graph should really be a primary goal. You should trim it down to only the relationships that are required for lookups and calculations.

             

             

             

            John Morina

             

            Pueblo System, Inc.

             

            CCQ-FM Inc.

             

            john@pueblo-systems.com

            1 of 1 people found this helpful
            • 3. Re: Advice on "where to now" - seperation vs single
              MicheleOlson

              Hi Katy,

               

              I use data separation about 95% of the time, but I also on occasion opt to keep multiple data files with tables that naturally go together for just the reason you mention in your next to last paragraph.

               

              I have found over time, this has given me the option to add modules as needed, by adding another data file with its tables without interfering with the remainder of the solution. This lets part of a the system to be in 'test' mode while the rest is in production and all use the same interface.

               

              I would likely go for the clean up of the "data files". Don't worry about the embarrassment of looking at the graphs of the older data files. You'll learn a lot by cleaning them out as well as be able to reduce redundancy and possibly speed up your overall solution. You will definitely have a clearer path for those who may come after you.

               

              Just my 2¢,

               

              Michele

              • 4. Re: Advice on "where to now" - seperation vs single
                kiwikaty

                Thank you Ibrahim, yes my UI file even with its 1000+ TO's is quite easy to navigate visually but the diagrams in the data files look like a kitten got loose with a ball of string. I think this is one of the reasons I wanted to ditch them !

                 

                Thank you John, I was thinking if I go for the "OR" option that once I had cleaned out the scripts and layouts (bar one for each table) that I could use Base Elements (which I find an invaluable tool) to look for "un-referenced" relationships and clear these out also. Due to the age of the system we have a lot of calcs that were made in the v5 days for use in relationships or for reports no longer in use but it is such a huge system now that it can be a bit scary culling out any schema items. It is hard not to worry that you have missed a dependency and I cringe at getting calls from our users about things that are suddenly "broken", so things tend to get bulleted but never removed!

                 

                Thank you to Michele also, all great advice. It would be nice to get the solution back into something more respectable as opposed to its current state of having the UI in both the data and UI file. I just was not sure of the best way forward for it. Your advice is greatly appreciated.

                • 5. Re: Advice on "where to now" - seperation vs single
                  Mike_Mitchell

                  Katy -

                   

                  I'll throw out a few ideas.

                   

                  1) I agree with the idea of cleaning out old layouts and scripts, especially layouts, as these will likely be contributing more to the size of the file. (Graphic objects do that.)

                   

                  2) I also agree with cleaning up the graphs. This will pay performance and maintenance dividends down the road. It will also reduce the likelihood of data corruption.

                   

                  3) The large number of calculation fields is a bit of a red flag. As you note, these are legacy from the pre-7 days and likely no longer needed. Your apprehension about removing them is understandable, and I appreciate it. I often cringe when cleaning up old systems too.

                   

                  I'd like to recommend a tool called BaseElements (http://www.goya.com.au/baseelements). It will take a Database Design Report (DDR) and break it down, telling you immediately what's still in use and what's not. It's worth getting rid of vestigial objects, including fields, because your system performance will improve dramatically if you do.

                   

                  HTH

                   

                  Mike

                   

                   

                  • 6. Re: Advice on "where to now" - seperation vs single
                    MicheleOlson

                    Mike_Mitchell wrote:

                     

                    I'd like to recommend a tool called BaseElements (http://www.goya.com.au/baseelements).

                    Second that recommendation!

                    • 7. Re: Advice on "where to now" - seperation vs single
                      kiwikaty

                      Yes I have BaseElements and would not be without it! It really is invaluable.

                      • 8. Re: Advice on "where to now" - seperation vs single
                        FileMakerProRocks

                        Hi Katy.

                         

                        One of my first thoughts would be: If you use container fields, migrate them to 'SuperContainer'. I was once approached by a FileMaker user who had accumulated a 90GB file over 8 years. With SuperContainer we brought it down to 50MB.

                         

                        This would be one step to start with.

                         

                        Greetings from Torbay.

                         

                        Cheers

                        Detlef

                        • 9. Re: Advice on "where to now" - seperation vs single
                          Malcolm

                          On 10/01/2012, at 2:50 AM, Mike_Mitchell said:

                           

                           

                          2) I also agree with cleaning up the graphs. This will pay performance and maintenance dividends down the road. It will also reduce the likelihood of data corruption.

                           

                          Note that you only need to retain relationships that are needed to generate data. The data file graph will usually have a few links to move data between tables.

                           

                           

                          3) The large number of calculation fields is a bit of a red flag. As you note, these are legacy from the pre-7 days and likely no longer needed. Your apprehension about removing them is understandable, and I appreciate it. I often cringe when cleaning up old systems too.

                           

                          Indexable calculation fields are simply converted to text/number/date/time/etc. Having changed the type, open the options and select auto-enter calculation. You'll find the original calculation sitting there. So don't be afraid, change them all to simple data type fields and mark them as deprecated in the comment field. 

                           

                          Malcolm

                          • 10. Re: Advice on "where to now" - seperation vs single
                            Mike_Mitchell

                            I appreciate Malcom's input about preserving the old calculation definitions. However, my point more along the lines of "I sympathize" rather than trying to preserve the old calculations. That's why I recommended BaseElements - so you can get rid of the old stuff with a measure of confidence.

                             

                            Just a point of clarification.       

                             

                            Mike

                            • 11. Re: Advice on "where to now" - seperation vs single
                              shearn

                              It's a big job, cleaning up that data file but since you already use BE, leverage it to reduce gotcha's you don't want your users to find. This will involve taking many BE snapshots and marking errors like "<field missing>" and "<relationship missing>". As you remove scripts, layouts and TO's, do it in small chunks and do a BE analysis after each one. If it finds more errors than you've already flagged, you know you've created a problem. Find out what you did and go back to the previous version or correct it in the current version and move on to the next chunk.

                              • 12. Re: Advice on "where to now" - seperation vs single
                                beverly

                                Tip: if I'm concerned about a calc I:

                                     1) convert it to "auto-enter" calc

                                or

                                     2) comment out the entire calc: /<<your calc here>>/

                                 

                                Neither of these remove the calculation (should you really need it as a calc and can revert), but they do cut the "calculating" over a distance.

                                 

                                Beverly Voth

                                • 13. Re: Advice on "where to now" - seperation vs single
                                  Malcolm

                                  Yes, I'm would be working toward that goal too but getting rid of stuff is hard and risky. It is much safer and easier to leave a lot of dead wood in the system than it is to start apologising for breaking things because you removed fields that didn't seem to have a purpose.

                                   

                                  When converting I think that the important task for the newly converted application is to ensure that the function of the systems are identical. Any changes, even optimisations, should be marked for the next version, unless they are a user requested priority. 

                                   

                                  Malcolm

                                  • 14. Re: Advice on "where to now" - seperation vs single
                                    CarstenLevin

                                    I am trying to make the decision whether to now make the UI file 1 big data AND UI file which would effectively consolidate the following files

                                    The main issue is to divide the soultion into data file(s) and ui file(s). But before merging data/table files into one single file you shuld consider whether there's anything to gain from merging them.

                                    In our solutions we would typically keep most of the tables in one file. Tables like a user table and maybe a table with containerfields with in-file-stored-files would have each their file.

                                    But when you are only having 5-15 tables there are also some advantages when keeping them in each their file. You will have simpler RD/TOG's.

                                    One thing though: I thing you should clean your data files from all extra elements: UI/layouts, scripts and TOG's thats not needed.

                                     

                                    Should I mention one disadvantage to having more files it is the issue of keeping accounts ajour. But with a user-table and a tiny bit of scripting this is easily centralized.

                                     

                                    And ... before doing anything: Have the files checked. Are they undamaged and intact?