7 Replies Latest reply on Jun 8, 2017 8:30 AM by fmpdude

    INSERT FROM URL Encoding Issue

    fmpdude

      Trying to pass a string as a GET parameter from FMPA 14 using INSERT FROM URL works fine for certain (read: simple) strings. But it doesn't appear that FMP correctly encodes more complicated strings that contain "?" and some other characters.

       

      For example: (?=^[0-5\\-]+$)(?!.*4321)\\d{3}-\\d{3}-\\d{4}

       

      Will not be correctly encoded by FMP and then received by the web service using INSERT FROM URL (with automatic encoding) and won't work.

       

      -----

       

      If I first encode this string in Java using the simple (one-liner) Encode method with UTF-8, I get this encoded string:

       

      %28%3F%3D%5E%5B0-5%5C-%5D%2B%24%29%28%3F%21.*4321%29%5Cd%7B3%7D-%5Cd%7B3%7D-%5Cd%7B4%7D

       

      This string, on the other hand, will work from FMP (after I first create it in Java, that is).

       

      So, I'm now wondering why FMP can't encode the string correctly to begin with. Can FMP only correctly encode "simple" strings?

       

      This INSERT FROM URL "Encoding" issue looks like a bug to me, or maybe this is where version 16 comes in to play?

       

      Just curious -- what does version 16 give you with Encoding the example RegEx above? Does it match the encoding I listed above?

       

      Thanks,

        • 1. Re: INSERT FROM URL Encoding Issue
          wimdecorte

          fmpdude wrote:

           

           

          For example: (?=^[0-5\\-]+$)(?!.*4321)\\d{3}-\\d{3}-\\d{4}

           

          his encoded string:

           

          %28%3F%3D%5E%5B0-5%5C-%5D%2B%24%29%28%3F%21.*4321%29%5Cd%7B3%7D-%5Cd%7B3%7D-%5Cd%7B4%7D

           

           

           

          GetAsURLEncoded ("(?=^[0-5\-]+$)(?!.*4321)\d{3}-\d{3}-\d{4}")

          in both 14 and 16 results in:

           

          (%3F%3D%5E%5B0-5%5C-%5D%2B%24)(%3F!.*4321)%5Cd%7B3%7D-%5Cd%7B3%7D-%5Cd%7B4%7D

           

          The difference with yours is that FM does not URLencode the ( and )

          • 2. Re: INSERT FROM URL Encoding Issue
            wimdecorte

            The other thing to watch out for is that little checkbox underneath the URL in the 'insert from URL' script step: it will by default do its own URL encoding, so if you give it something that you have already encoded it will do it again, creating a bit of a mess.  So remember to turn it off if you are feeding it a URL that is already encoded.

             

            2017-06-08_07-54-12.png

            1 of 1 people found this helpful
            • 3. Re: INSERT FROM URL Encoding Issue
              wimdecorte

              wimdecorte wrote:

               

              fmpdude wrote:

               

               

              For example: (?=^[0-5\\-]+$)(?!.*4321)\\d{3}-\\d{3}-\\d{4}

               

              his encoded string:

               

              %28%3F%3D%5E%5B0-5%5C-%5D%2B%24%29%28%3F%21.*4321%29%5Cd%7B3%7D-%5Cd%7B3%7D-%5Cd%7B4%7D

               

               

               

              GetAsURLEncoded ("(?=^[0-5\-]+$)(?!.*4321)\d{3}-\d{3}-\d{4}")

              in both 14 and 16 results in:

               

              (%3F%3D%5E%5B0-5%5C-%5D%2B%24)(%3F!.*4321)%5Cd%7B3%7D-%5Cd%7B3%7D-%5Cd%7B4%7D

               

              The difference with yours is that FM does not URLencode the ( and )

               

              Spotted one more that FM does not URL encode: the !

              Not a bug as far as I can tell.  RFC 1738 specifies what characters are allowed un-encoded in URLs:

               

              Thus, only alphanumerics, the special characters "$-_.+!*'(),", and

                reserved characters used for their reserved purposes may be used

                unencoded within a URL.

              1 of 1 people found this helpful
              • 4. Re: INSERT FROM URL Encoding Issue
                fmpdude

                Hey Wim,

                 

                Thanks for your reply.

                 

                Unless I missed something from your posting, unfortunately, the two encoded strings don't match and the GetAsURLEncoded (though initially promising) doesn't work.

                 

                (?=^[0-5\-]+$)(?!.*4321)\d{3}-\d{3}-\d{4}

                 

                GetAsURLEncoded Returns:

                 

                (%3F%3D%5E%5B0-5%5C-%5D%2B%24)(%3F!.*4321)%5Cd%7B3%7D-%5Cd%7B3%7D-%5Cd%7B4%7D

                 

                Web Service Returns (since it can't resolve the encoding):

                 

                <html><body><h2>404 Not found</h2></body></html>

                 

                ----

                 

                Java UrlEncoded String returns:

                 

                %28%3F%3D%5E%5B0-5%5C-%5D%2B%24%29%28%3F%21.*4321%29%5Cd%7B3%7D-%5Cd%7B3%7D-%5Cd%7B4%7D

                 

                (works.)

                 

                There could be nested parentheses or other characters so as I understand it, there isn't a solution yet in FMP.

                 

                A quick romp into Beyond Compare confirms what you stated: FMP doesn't correctly encode the parenthesizes or "!", and possibly other characters (below, FMP encoding top, Java encoding bottom):

                 

                ---

                I tried with both auto-encode off and on in INSERT FROM URL.

                • 5. Re: INSERT FROM URL Encoding Issue
                  wimdecorte

                  fmpdude wrote:

                   

                  Hey Wim,

                   

                  Thanks for your reply.

                   

                  Unless I missed something from your posting, unfortunately, the two encoded strings don't match

                   

                  They don't.  And it seems to be a difference in interpretation of what is allowed in the URL.  Your java code for instance encodes ( and ) and ! and $ but not the -

                  Looks like the web service is ok with that.

                   

                  FM encodes the $ but not the ( and ) and -, and that looks to be acceptable by RFC standards except that the web service expects something else.

                   

                  fmpdude wrote:

                   

                   

                  There could be nested parentheses or other characters so as I understand it, there isn't a solution yet in FMP.

                   

                   

                  There is: do the explicit encoding yourself through Substitute().  That's not uncommon when interacting with web services.  I find myself doing the same when I do ASP.NET coding in C#.  Unless you have control over the web service  you have to live with its quirks and quirks of the tool that you use.

                   

                  If only all tools and platforms were a little more strict in following guidelines and standards, how much easier our lives would be in the integration space...

                  1 of 1 people found this helpful
                  • 6. Re: INSERT FROM URL Encoding Issue
                    fmpdude

                    I wrote the WS so I have control over it, but I don't know a way to just pass a GET string parameter like this as a literal string that won't be messed with on the way (other than encoding).

                     

                    I could still use JDBC to read that field and UPDATE it with the correct Java-encoded string, but that would be a bit of work.

                     

                    Thanks for your replies and suggestions!

                    • 7. Re: INSERT FROM URL Encoding Issue
                      wimdecorte

                      Why not make it a POST and pass the string as a payload, in JSON or otherwise?  That would get you around the URL encoding.

                      • 8. Re: INSERT FROM URL Encoding Issue
                        fmpdude

                        I was all ready to start adding nested Substitute function calls to my GetAsURLEncoded function call, but, guess what? The problem turned out to be that I had forgotten to update my script to use the new field I created to hold the GetAsURLEncoded-encoded string!  

                         

                        GetAsURLEncoded works perfectly for every case I've tested.

                         

                        -----------------

                         

                        FWIW, I found that I still needed the auto-encoding URL In the INSERT FROM URL or it wouldn't work in all cases.

                        -----------------


                        Here's the updated screenshot showing every RegEx I can throw at the WS correctly receiving and returning to FMP the result.

                        Thanks Wim, you're the best.