14 Replies Latest reply on Sep 21, 2016 5:37 AM by wfgclapp

    Timeout when using RESTfm POST

    wfgclapp

      I am looking to create records in a hosted FileMaker file using Goya's RESTfm.

       

      The problem I'm having seems to be on FileMaker or my server's end, so I'm hoping for some CWP expertise (something I have none of).

       

      I am using RESTfm to POST a bulk set of records, all contained in a file. Data formatted as JSON.

       

      Basically, it works. I can POST my JSON file, and records are then created in FileMaker as they are supposed to be.

       

      The problem is that I am needing to post around 11,000 records and the most that ever get posted with my process is around 1,500. BUT IT VARIES. One minute I'll get 1,000 recs through, the next only 600. The next 950, the next 590, or whatever. This is a testing situation I'm posting the same file of data each time, and between tries I'm clearing the FM table.

       

      So, it doesn't seem to be hung on a particular record. Seems like a timeout issue.

       

      But I don't know if this is a FMS issue or a Windows issue (I am running FMS15 on a Windows 2008 server).

       

      I never get a response back via RESTfm, so no error codes. It's like the process just dies.

       

      I could probably break my process down into sending many requests of smaller records, but I first want to see if there are any settings or such that I can adjust to get the largest number of records POSTd in the same request.

       

      I've never fooled with FileMaker's CWP before.

       

      Anyone have a thought, or some good references or documentation to point me towards?

       

      Thanks much.

        • 1. Re: Timeout when using RESTfm POST
          mikebeargie

          I would highly suggest that you upload a file instead and then use FileMaker's native import.

           

          Trying to pass a JSON array with that many records probably crashing the memory buffer when the FMS side tries to assemble the post request. You either need to batch it on your web side into batches of 50 or so records at a time and then process those chunks one at a time; or upload the file (XML is probably best) and then have filemaker execute a script that runs an import on that file.

           

          You wouldn't get an error back in this case, essentially the server is working itself to death and REST really isn't a poling language where it's checking back with progress reports as it goes.

          1 of 1 people found this helpful
          • 2. Re: Timeout when using RESTfm POST
            deninger

            Are you using a BULK POST? According the the Support Pages for RESTfm:

             

            Support for bulk operations was first added in version 3.

            Bulk operations allow multiple operations of the same type (Create, Read, Update or Delete) to be bundled in the one HTTP request-response transaction. This can provide significant speed advantages where each operation is quick, but there are many records and the cumulative latency of multiple single HTTP transactions is slowing things down.

            All bulk operations act on the same 'bulk level' URI: /RESTfm/{database}/bulk/{layout}

            Note: Bulk operations will still be limited by web server/client timeouts, thus limiting the total number of operations that can reasonably be handled in a single transaction.

             

            URI /RESTfm/{database}/bulk/{layout} - Bulk Operations | restfm.com

             

            If you are not, you might try it instead. If you already are, you might look for ways to configure your web server and client timeouts, though I am not sure this will give you enough wiggle room...

            • 3. Re: Timeout when using RESTfm POST
              wfgclapp

              deninger,

               

              yes, I am using the BULK operation.

               

              I'm not sure where all my web server timeout settings are, but I'll poke around. THanks!

              • 4. Re: Timeout when using RESTfm POST
                wfgclapp

                Thanks mike.

                 

                I've been uploaded delim files and importing but this is what I am trying to get away from. Doing it that way, I have to have an extra layer of control on both ends so to keep a new file from being created until the old one has been processed, etc. Was really hoping RESTfm would be the way to go. Maybe it's not.

                • 5. Re: Timeout when using RESTfm POST
                  mikebeargie

                  A lot of APIs out there are severely limited in per-request throughput. Most will only take interactions with 1-10 records/requests at a time.

                   

                  Honestly if I was a web developer and someone asked why my API didn’t handle 11,000 records in one request, I’d probably ban their account for trying to DDoS my server.

                   

                  APIs are usually designed to be transactional for smaller sets of records. Larger sets of records are still being handled with file generation or shadow tables.

                  2 of 2 people found this helpful
                  • 6. Re: Timeout when using RESTfm POST
                    wfgclapp

                    Mike Beargie wrote:

                     

                    A lot of APIs out there are severely limited in per-request throughput. Most will only take interactions with 1-10 records/requests at a time.

                     

                    APIs are usually designed to be transactional for smaller sets of records. Larger sets of records are still being handled with file generation or shadow tables.

                    Fair enough, I didn't realize that. I sure do hate dealing with those delimited files. 

                    You've probably saved me a bunch of time. thanks!

                    • 7. Re: Timeout when using RESTfm POST
                      deninger

                      There is another option, though it may not solve your specific problem.

                       

                      I am using RESTfm to update records in bulk as well. But instead of adding records in bulk, I created a custom .php file that allows me to upload the entire JSON batch to the server in a single field. Then I call a script from the RESTfm API when it is done to "unpack" it into the necessary records.  To avoid a time-out issue, I call the script as PSOS so that the RESTfm API returns the result (custom from the .php file I am using) to indicate success. Here success is just the upload of the data and not the unpacking. That progress is logged by the script unpacking the data.

                       

                      This might work as the upload itself of the JSON will possibly be under the timeout period.

                      • 8. Re: Timeout when using RESTfm POST
                        user19752

                        PHP setting is in

                        C:\Program Files\FileMaker\FileMaker Server\Web Publishing\publishing-engine\php\php.ini

                         

                        I saw default values as

                        max_execution_time = 30 ; Maximum execution time of each script, in seconds

                        max_input_time = 60    ; Maximum amount of time each script may spend parsing request data

                        memory_limit = 128M  ; Maximum amount of memory a script may consume (8MB)
                        • 9. Re: Timeout when using RESTfm POST
                          fmpdude

                          Have you ever thought of just hosting your own REST service in Java using an API like JAX-RS? I do this all the time and it's quick, reliable, usable from other clients (no expensive FMS needed, for example), and, best yet, all the JAX-RS REST stuff is .... FREE.

                           

                          One of my REST services I use from FileMkaker receives via INSERT FROM URL (POST) a list of all the values of one field for a current year. With POST, there's no hard limit to how much data you can upload.

                           

                          HOPE THIS HELPS.

                          • 10. Re: Timeout when using RESTfm POST
                            wfgclapp

                            user19752 wrote:

                             

                            PHP setting is in

                            C:\Program Files\FileMaker\FileMaker Server\Web Publishing\publishing-engine\php\php.ini

                             

                            I saw default values as

                            max_execution_time = 30 ; Maximum execution time of each script, in seconds

                            max_input_time = 60 ; Maximum amount of time each script may spend parsing request data

                            memory_limit = 128M ; Maximum amount of memory a script may consume (8MB)

                            those are script execution timeouts. what about URL calls?

                            • 11. Re: Timeout when using RESTfm POST
                              wfgclapp

                              deninger wrote:

                               

                              There is another option, though it may not solve your specific problem.

                               

                              I am using RESTfm to update records in bulk as well. But instead of adding records in bulk, I created a custom .php file that allows me to upload the entire JSON batch to the server in a single field. Then I call a script from the RESTfm API when it is done to "unpack" it into the necessary records. To avoid a time-out issue, I call the script as PSOS so that the RESTfm API returns the result (custom from the .php file I am using) to indicate success. Here success is just the upload of the data and not the unpacking. That progress is logged by the script unpacking the data.

                               

                              This might work as the upload itself of the JSON will possibly be under the timeout period.

                              Hmmm....this sounds very promising! Hopefully I can try this out soon. Thanks!

                              • 12. Re: Timeout when using RESTfm POST
                                wfgclapp

                                fmpdude wrote:

                                 

                                Have you ever thought of just hosting your own REST service in Java using an API like JAX-RS? I do this all the time and it's quick, reliable, usable from other clients (no expensive FMS needed, for example), and, best yet, all the JAX-RS REST stuff is .... FREE.

                                 

                                One of my REST services I use from FileMkaker receives via INSERT FROM URL (POST) a list of all the values of one field for a current year. With POST, there's no hard limit to how much data you can upload.

                                 

                                HOPE THIS HELPS.

                                No, not really. I don't really have the skills for that I don't think. Good thought though, thanks.

                                • 13. Re: Timeout when using RESTfm POST
                                  fmpdude

                                  Check out the tutorials. Unlike FileMaker, there are TONS of current books. The O'Reiley JAX-RS book is excellent and takes you step by step.

                                   

                                  Java is a simple language and you can do ANYTHING with it from a simple "Hello World" to creating data structures as complicated as you can envision.

                                   

                                  Whatever you do, I hope you solve your problem soon.

                                   

                                  Good luck!

                                  • 14. Re: Timeout when using RESTfm POST
                                    wfgclapp

                                    fmpdude wrote:

                                     

                                    Check out the tutorials. Unlike FileMaker, there are TONS of current books. The O'Reiley JAX-RS book is excellent and takes you step by step.

                                     

                                    Java is a simple language and you can do ANYTHING with it from a simple "Hello World" to creating data structures as complicated as you can envision.

                                     

                                    Whatever you do, I hope you solve your problem soon.

                                     

                                    Good luck!

                                    thanks. i'm certainly interested.