11 Replies Latest reply on Mar 28, 2017 2:56 AM by wimdecorte

    Is there a problem with an XML API request often returning truncated XML?

    douglerner

      Our server is making requests for XML of the form:

       

      https://xxx.somedomain.com/fmi/xml/fmresultset.xml?-db=FM%20OurDB%2014&-lay=XML_Trans&User_Page=http:%2F%2…

       

      The call is supposed to return XML ending in </fmresultset>. But oftentimes the XML returned is not complete; it gets truncated before finishing the call.

       

       

      I did a test where I called that URL 20 times and just displayed how many characters are returned:

       

       

       

      Here are the results. 8762 is the correct number, but in 15 out of 20 calls only 7915 characters are returned - truncated XML.

       

       

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 8762

      character returned: 8762

      character returned: 8762

      character returned: 7915

      character returned: 8762

      character returned: 8762

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

      character returned: 7915

       

       

      It's very regular. Either 8762 characters are turned with a final <fmresultset> or 7915 characters are returned and the XML is truncated before the end.

       

      Anybody ever see any problem like this before? Any suggestions?

       

      Thanks,

       

      doug

        • 1. Re: Is there a problem with an XML API request often returning truncated XML?
          user19752

          What is your action, -find ?

          Where the truncation occurs, middle of field data?

          Are there any unstored field in return?

           

          Do you have same issue if the correct result is shorter than 7915? (or 8192)

          • 2. Re: Is there a problem with an XML API request often returning truncated XML?
            douglerner

            Thanks for your response. It turns out to be a problem with our SSL https outbound request function. If I use http instead of https the results are always correct. So we are attempting to fix the SSL on our server side.

             

            Thanks,

             

            doug

            2 of 2 people found this helpful
            • 3. Re: Is there a problem with an XML API request often returning truncated XML?
              wimdecorte

              Looking at the syntax of the XML query; you explicitly mention a fully qualified field name "members::user_page".  Are you doing a search on a related field?

              • 4. Re: Is there a problem with an XML API request often returning truncated XML?
                douglerner

                It is searching on that related field, yes. That's what the lookup is being done on. I believe (I didn't create the database script) that the field itself is one of those that gets evaluated when the record is added and isn't refreshed all the time at run time.

                 

                What we have learned so far in our testing though is this:

                 

                We have discovered that the error so far is only replicable with outbound https requests from our server (a special kind of server) to the FM server. We used the exact same code to make https requests to other non-FM https servers - doing hundreds of calls one after another - and the https never fails when calling the other servers.

                 

                On the other hand, if we call the FM server from a C# test on a separate computer, or by directly going to the API URL in Chrome, or in Firefox then the FM server always returns the same, complete results.

                 

                So it seems to be something particular with the connection between just our special server's outbound https requests and the FM server. In a case like this, it's difficult to say whose "fault" it is.

                 

                But, since other clients (the C# program, Chrome, Firefox) are able to connect to the FM server always successfully it seems we have to try fix the problem on our end, unless there is something tweakable about the FM server to make it act like all the other servers we are able to make requests to just fine.

                1 of 1 people found this helpful
                • 5. Re: Is there a problem with an XML API request often returning truncated XML?
                  wimdecorte

                  douglerner wrote:

                   

                   

                   

                  But, since other clients (the C# program, Chrome, Firefox) are able to connect to the FM server always successfully it seems we have to try fix the problem on our end, unless there is something tweakable about the FM server to make it act like all the other servers we are able to make requests to just fine.

                   

                  I use the XML API a lot, either directly or through fmDotNet (when I write integrations in C#).  Testing tools I use are Fiddler on Windows and Postman on Mac.  I've never seen FMS reply with truncated XML so I think you are right that it is something that needs to be tweaked on the 'special server' side.

                   

                  In the http response you get, what response code do you receive?

                  • 6. Re: Is there a problem with an XML API request often returning truncated XML?
                    douglerner

                    Whether truncated or not the initial headers are always the same, like:

                     

                    <?xml version ="1.0" encoding="UTF-8" standalone="no" ?><!DOCTYPE fmresultset PUBLIC "-//FMI//DTD fmresultset//EN" "https://someserver.somedomain.com/fmi/xml/fmresultset.dtd">

                     

                    The error code is always zero:

                     

                    <error code="0"/>

                    <product build="4/7/2016" name="FileMaker Web Publishing Engine" version="15.0.1.137"/>

                     

                    Between the <metadata>...</metadata> tags are always the fields returned.

                     

                    The resultsetcount tag is always correct, for example:

                     

                    <resultset count="3" fetch-size="3">

                     

                    Then there are the <record>...</record tags>

                     

                    At the end, when correct, there is

                     

                     

                    </resultset>

                     

                    </fmresultset>

                     

                    But when the https fails (http hasn't failed yet) those last two tags are missing and the XML is truncated too early - and almost always in the same place, sometimes working and sometimes not.

                     

                    If we make the call with http instead of https though, so far it works every time.

                     

                    doug

                    • 7. Re: Is there a problem with an XML API request often returning truncated XML?
                      wimdecorte

                      douglerner wrote:

                       

                      Whether truncated or not the initial headers are always the same, like:

                       

                      <?xml version ="1.0" encoding="UTF-8" standalone="no" ?><!DOCTYPE fmresultset PUBLIC "-//FMI//DTD fmresultset//EN" "https://someserver.somedomain.com/fmi/xml/fmresultset.dtd">

                       

                       

                      I was talking about the HTTP response headers and the HTTP response code.  Is it 200 - OK?

                      • 8. Re: Is there a problem with an XML API request often returning truncated XML?
                        douglerner

                        I see what you mean. I'll check and find out. I need to do some extra logging to confirm that.

                         

                        doug

                        • 9. Re: Is there a problem with an XML API request often returning truncated XML?
                          douglerner

                          OK, I checked. "HTTP 200 OK" is consistently returned in the reply from the FM server.

                           

                          doug

                          • 10. Re: Is there a problem with an XML API request often returning truncated XML?
                            douglerner

                            We discovered the cause, and made a fix. It turned out to be a combination of the way we made our httpReq and how FM returns the API calls, which appears to be different from the way all the other servers we tested return requests.

                             

                            We have 3 ways to tell that an http GET request is done. We label them in our server as follows:

                             

                            - ok1: got all chunk+trailer

                             

                            -ok2: Content-length or just EOF demarks end of body data

                             

                            -ok3: Nothing more is coming in, so wrap up what we've got

                             

                            All the other sites we tested with end with ok2, but FM ends with ok3.

                             

                            However, more detailed checking of the raw TCP/I data shows that FM is using chunked transfer while all the other sites are just specifying content-length.

                             

                            So it's logical to see working sites ending with ok2 (since content-length is known) but FM replies in  chunks + trailer.

                             

                            But the strange thing was that while the response headers grabbed from curl/wget/openssl consistently show the "Transfer-Encoding: chunked" line, it was absent in the returns we received when we made the call to FM from our server. With no "Content-length" and no "Transfer-Encoding: chunked" we were falling back to "wait and see mode" and so when no more data was the available, http request is closed with "ok3"

                             

                            While looking for why FM server doesn't send the "Transfer-Encoding: chunked" line we examined our own request header more. It was HTTP/1.0. In other words, we were saying to the FM server we don't handle chunked pages, even though we do. We confirmed the same behavior, using openssl to send a GET HTTP/1.0 header to FM server. In this case as well no "Transfer-Encoding: chunked" was sent back by the FM server.

                             

                            Our solution? We modified our outgoing https header string and set it to HTTP/1.1. Now 100% of the requests we made (ie hundreds) passed the test. The http request now ends with ok1 as expected.

                             

                            So that ends that, as it were.

                            3 of 3 people found this helpful