1 2 Previous Next 20 Replies Latest reply on Jan 20, 2015 2:00 PM by gdurniak

    Can filemaker pro handle a database for an online university with 6 million student records ?

    binu.alexander

      Just want to know what is the largest tested database with FM Pro 13.  I read somewhere that it can handle upto 8TB for a single file.  What about performance  at  really high records    and  large number of users ?

        • 1. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
          gdurniak

          It all depends,  and is case by case

           

          Out largest Table has 13 Million records, so it can work,  but what is a "large number of users" ?  We have about 50

           

          Also,  if thousands need web access, it may be a bit sluggish

           

          greg

          • 2. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
            Mike_Mitchell

            When you start dealing with the high end of loads like this, you're going to have to be very thorough about optimizing both the database and the PHP code. And it still will likely not be as "snappy" as sites based on other technologies.

             

            Nevertheless, it's possible.

            • 4. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
              mbraendle

              To be clear: Without clear specification what the system should do, we can't answer your question. Specification of all the tasks involves speaking with the different interests groups, collecting their requirements, evaluating and reviewing the requirements, writing a report, evaluating the specifications of possible software solutions and matching them to the specifications of the system, finally making a proposal and a decision with key persons.

              This may be a process that takes several weeks.

               

              The key term here is "online university": Assume that your 6 million users are distributed world-wide and access the system at least once per day via the Web, which is 250'000 accesses per hour, or 70 accesses per second.

              You need a distributed infrastructure to support such a work-load. FileMaker does not support distributed servers.

              That's why I said no.

              • 5. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                Mike_Mitchell

                If we were talking about 6 million users, I would absolutely agree. Except he said 6 million records. That's not the same thing.

                 

                So you're absolutely correct: The number of users is really what drives the question. The number of records is far less important. FileMaker can handle a lot of records, especially if everything can be indexed properly. But if you're talking about 5,000 users simultaneously hitting the server, you're done.

                 

                Therefore, you're absolutely correct (again), when you say:

                 

                "Without clear specification what the system should do, we can't answer your question."

                 

                 

                • 6. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                  Mike Duncan

                  Can a FileMaker database handle that many records? Yes, it is possible and within the range of it's specifications.

                   

                  The question is a bit open ended however. This does not mean you can design it any way and still expect performance. That goes for any system, any platform or language. You can design a big old mess in just about anything

                  • 7. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                    chris.schmitz

                    I don't know that I would suggest using FIleMaker for the data storage of that many records. If you have it all in one file it's going to make it hell to backup and work with, even with progressive backups. e.g. Ever want to work on a local copy of the file? Get ready for a hell of a move!

                     

                    That said, there are still options you can go with for using FileMaker as a front end interface while storing the data in a different database that is a bit better at storing a ton of records.

                     

                    The first and best option that comes to my mind is using ESS (External SQL Source) to connect your filemaker front end to your version specific (see the documentation) oracle|microsoft sql| mysql database. Using ESS allows you to interact with the data and build relationships the same way (for the most part) as you would with native FileMaker database. My suggestion would be a FileMaker front end with a MySQL back end. The setup is easy, mysql is free, and it's probably the most popular opensource database so there are tons of resources to refer to as far as setup and usage.

                     

                    You can also use an ODBC driver to communicate back and forth between FileMaker and your database. This has the advantage of offering more databases to work with as it doesn't have the same restrictions that ESS does, but it's not going to be as fluid of an experience.

                    • 8. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                      binu.alexander

                      Thank you guys

                       

                      The number of concurrent users at a single  point in time would be between 1000--2000

                       

                      Is it possible to use WebDirect or would I have to design in PHP ?

                       

                      If Filemaker Pro can handle it and give good results  then  I don't have to look at other solutions.

                       

                      Also since I don't have expertise in SQL  ,  I just wanted to know if I could do it solely in FM Pro .

                      • 9. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                        Mike_Mitchell

                        I think it might be important to keep in mind that the original question was, "Can FileMaker do this?" The answer to that question is, "Well, depends on use case."

                         

                        The rest of the story is, "Is FileMaker the best choice for this application?" I think it might be safe to say, "Well, maybe not." FileMaker is, always has been, and is clearly intended to be, a workgroup database application. True, the size of the "workgroup" tends to grow larger every year, but nevertheless, it was never intended to be an enterprise application.

                         

                        So to answer the OP's question, "Can FileMaker do this?", we can say, "Quite possibly, if you're careful and you mind the limitations." I think most of use would probably ask the OP in response, "Are you really sure FileMaker is the best choice for this application?" I'm sure he has reasons to ask the question - perhaps he has an existing infrastructure he wants to use, or perhaps he wants to connect to some existing FM databases, or perhaps he has existing expertise he wants to leverage. But he will - as some have pointed out - have to be very careful and mind the various preconditions and setup constraints associated with larger deployments.

                        • 10. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                          Mike_Mitchell

                          WebDirect is definitely a no-go. You would need to use the PHP API.

                           

                          And concurrent users of 1000 or more will likely overload the server. The FMS process is basically single-threaded and cannot load-balance, so if you were to have, say, 500 users all hit the server simultaneously with a request, they will be looking at a spinning icon for some time as the server tries to clear the backlog. How successful you would be would depend on how active the site would be (i.e., how often users would submit requests), what those requests would need from the database, how much server horsepower you have, and how well the site and database were optimized.

                           

                          But I would be reluctant to try it myself. It's right at the razor's edge, IMO.

                          • 11. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                            DavidZakary

                            My biggest database right now is just under 38 million records. The system does an automated import every morning and adds between 40 and 50,000 records. There are very few concurrent users and most are local to the city. Right now that 38 million record file is under 7 gigs and could be smaller if we cleared out some legacy garbage that is taking up extra index space. Backing up isn't really an issue. A quick test with the Verify Backup Integrity on takes about 4 minutes. With Verify Backup Integrity off (not recommended) it takes 55 seconds.

                             

                            Most operations are very fast, including generating Javascript-based chats in a web viewer and grabbing summary information via ExecuteSQL for display on a dashboard interface. Even over LTE on my iPhone I can get more than satisfactory results. Most users are coming in, viewing the dashboard and getting out. This system is designed for data analysis, no data entry by users other than to generate reports. Data entry is via the daily server script to do the importing. Importing those 40-50,000 records takes about 3 minutes.

                             

                            Other operations can be glacial. If we need to change the database schema for instance. Adding a calculated field to the main data table is painful. Searches generally aren't bad. At one point there was a summary field on a layout and it was VERY bad. It's long gone. Working with smaller record sets helps a lot. We found that most users were interested in what happened in the past week or 30 days 90% of the time. Doing a search to limit those 38 million records down to under 1 million also sped things up considerably.

                             

                            I would go with the phrase - your mileage may vary. Depending on what you're trying to do, an optimal data model and record set, user load, network configuration you may file FileMaker works perfectly. You may also find that it doesn't suit your particular needs.

                             

                            If you've got users logging in to work with a very small set of records you'd likely find things work pretty well. If you're sorting and summarizing huge data sets with lots of network traffic you may not like what you get. FileMaker doesn't do any load balancing or mirroring of files so everything is going to come down to a few machines.

                             

                            You're likely not going to use WebDirect. If you've got 1000-2000 concurrent users WD isn't going to be able to handle that at this time. PHP connecting to a FileMaker back end is more sensible. If you're doing PHP you're in the realm of double-developing - building both a FileMaker back-end and the PHP-based front-end. Any system where you're using technology A to connect to database B is going to have that double development issue, doesn't matter if its FileMaker or an SQL-based system. A purely native FileMaker system will be less development.

                            • 12. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                              chris.schmitz

                              Mike_Mitchell wrote:

                               

                              I think it might be important to keep in mind that the original question was, "Can FileMaker do this?" The answer to that question is, "Well, depends on use case."

                               

                              ... True, the size of the "workgroup" tends to grow larger every year, but nevertheless, it was never intended to be an enterprise application.

                               

                              So to answer the OP's question, "Can FileMaker do this?", we can say, "Quite possibly, if you're careful and you mind the limitations." I think most of use would probably ask the OP in response, "Are you really sure FileMaker is the best choice for this application?" ...

                              Good point and I agree.

                              • 13. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                                wimdecorte

                                Mike_Mitchell wrote:

                                 

                                The FMS process is basically single-threaded and cannot load-balance, so if you were to have, say, 500 users all hit the server simultaneously with a request, they will be looking at a spinning icon for some time as the server tries to clear the backlog.

                                 

                                That's not completely accurate; FMS is fully multi-threaded so the more processors and cores you give it the more concurrent requests it can handle.  Whether there is a backlog depends on the number of requests that come in, how long they take to complete and how many resources are available to complete them.

                                 

                                Load-balancing is a term that applies to having multiple servers working together.  FMS can not do that.

                                • 14. Re: Can filemaker pro handle a database for an online university with 6 million student records ?
                                  jrenfrew

                                  Pause.

                                   

                                  Get a copy of the FileMaker training series.

                                  Read it thoroughly.

                                   

                                  Pause a bit more.

                                  Read it some more.

                                  1 2 Previous Next