1 2 Previous Next 17 Replies Latest reply on Oct 4, 2016 5:24 PM by NickLightbody

    FTP uploads from Filemaker

    NickLightbody

      I am currently polishing a solution, originally it was just a small idea, which has appeared quite useful, into an App.

       

      It is called rwApp and it generates a Responsive Web site from simple textual input, with some use of Markdown and a few codes.

       

      I have explained what it is and why here

       

      http://www.deskspace.com/rwapp_features.html

       

      In refining it I have created a few sites recently which give an idea of what it can do:

       

         http://www.deskspace.com

         http://www.nicklightbody.com

         http://www.deskspace.com/t/ionian/index.html

         http://www.bolnore.com/t/beijing/index.html

         http://www.bolnore.com/t/trialno1/

       

      The last is made entirely on an iPhone.

       

      If anyone wishes to volunteer to review the rwApp, and get a free license after release, please let me know. You can email me on nick.lightbody@deskspace.com

       

      The challenge was how to ftp the new pages up to the web server automatically from OS X 10.11. I had been manually dragging them into an ftp window and over Sunday lunch my wife suggested that this really was not good enough!

       

      A little searching showed several people (bigtom springs to mind) had shared their knowledge of doing uploading with ftp with an expression creating an Applescript which itself used Curl to achieve the ftp - but after several hours effort I couldn't get the upload to work.

       

      Then I found a brilliant thread

       

      http://comments.gmane.org/gmane.comp.web.curl.general/5634

       

      which provided the syntax for Curl to report back all its environmental values on receipt of the command, which revealed that a $1 within the ftp password was disappearing within the Applescript, hence my authentication was failing. So I changed the password to not inc a $ character and it worked as expected. I imagine I could have escaped it somehow?

       

      Here is the brilliant Curl command to be run in Terminal:

       

      A Terminal Command $               curl --trace -n -v -ssl -l ftp://<username>:<password>@<svr IP address>/<folder>/<file.html>

       

      This gave the following report back to help me solve the issue - with explicit details of the username and password received:

       

      Warning: -v, --verbose overrides an earlier trace/verbose option

      *   Trying 193.xxx.xxx.xxx..

      * Connected to 193.xxx.xxx.xxx (193.xxx.xxx.xxx) port 21 (#0)

      < 220 ProFTPD 1.3.5a Server (ProFTPD) [193.xxx.xxx.xxx]

      > USER xxxxxxxx-ftp

      < 331 Password required for xxxxxxxx-ftp

      > PASS abcdef

      < 530 Login incorrect.

      * Access denied: 530

      * Closing connection 0

       

      In this case the password input would have been abc$1def and only the abcdef survived the Applescript.

       

      The final Applescript looked like this:

       

      do Applescript curl -T '/private/var/folders/65/g9tpm0qs5tg1rbxf6s25z2wm0000gn/T/S10/index.html' ftp://user-ftp:password@193.xxx.xxx.xxx/httpdocs/index.html

       

      The expression itself is of course in quotes, hence the use of single quotes around the source file.

       

      So I was just exporting my string as an html file to get(temporary path) and then immediately using curl to ftp from that temp folder to the web server.

       

      Very fast.

       

      The only point to watch, besides the $ business, is that there were some comments that this may work better using the server IP rather than the domain name, so that is what I did, logically an IP should be better I would think since it is a real address rather than a user friendly label which is all a domain name is.

       

      Cheers, Nick

        • 1. Re: FTP uploads from Filemaker
          monkeybreadsoftware

          I would still always prefer to use CURL via Plugin instead of command line to avoid trouble with parsing all the output and having more control over the transfer.

          And for security I would recommend using FTPS or SFTP.

          • 2. Re: FTP uploads from Filemaker
            bigtom

            I am with monkeybreadsoftware on this one about secure FTP. Although I think you can run some sort of secure FTP with the cURL command built into OSX. I cannot remember for sure though.

             

            NickLightbody One of the nice things about the OSX command is that you can simultaneously upload a number of files which really speeds things up if you have the bandwidth. From what I have seen with plugins you have to do files one at a time or at least make numerous calls to the script and make several connections.

             

            I am in for reviewing the app. I will send you an email.

            • 3. Re: FTP uploads from Filemaker
              user19752

              You need username and password to be urlencoded, so change $ to  %24.

               

              I forgot that % should be or not encoded in applescript command line.

              • 4. Re: FTP uploads from Filemaker
                monkeybreadsoftware

                MBS Plugin can do asynchronous transfers.

                a 100 in parallel if you need.

                 

                And when a transfer finishes, you get a scrip triggered.

                • 5. Re: FTP uploads from Filemaker
                  NickLightbody

                  Hi Christian, BigTom and User19752

                   

                  Thanks for the comments.

                   

                  Tom, thanks for offering to review, you are on the list.

                   

                  The user inputs are as shown on attached screen shot. The ftp protocol can be changed to ftps or sftp as wished. I have not tried either yet and I don't know what this server is set up for. When I do I shall try to remember to report back.

                   

                  The parsing of the applescript was only tricky because I didn't realise that the $ would create an issue, otherwise it does not seem a complex string to create or automate.

                   

                  Because the rwApp is creating small efficient html pages, potentially many pages, my experience so far of the upload speeds has been very good but I will add a log so the stats can be analysed in the future. I just created a single web page for the whole of the Holy Bible, the biggest text source I could find, and rwApp uploaded its 17mb html file in about 45 seconds, so I don't see speed as any sort of problem.

                   

                  You can find it here

                   

                  http://www.bible.com/bible_index.html

                   

                  I have done a short YouTube video on the Holy Bible in a single page which is here

                   

                  rwApp TheWholeBible - YouTube

                   

                  Cheers, Nick

                   

                  rwApp_settings.png

                  • 6. Re: FTP uploads from Filemaker
                    NickLightbody

                    As an added convenience for anyone finding this thread and wishing to use this technique here is some points on encoding etc best resolved, I would suggest, as follows:

                     

                    In using CURL you are inputting the following data

                     

                    1. IP address of the ftp server

                    2. html page name

                    3. account / user name

                    4. password

                     

                    According to RFC 3986, the characters in a URL have to be taken from a defined set of unreserved and reserved ASCII characters. Any other characters are not allowed in a URL.

                     

                    The unreserved characters can be encoded, but should not be encoded. The unreserved characters are:

                     

                    A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

                    a b c d e f g h i j k l m n o p q r s t u v w x y z

                    0 1 2 3 4 5 6 7 8 9 - _ . ~

                     

                    The reserved characters have to be encoded only under certain circumstances. The reserved characters are:

                     

                    ! * ' ( ) ; : @ & = + $ , / ? % # [ ]

                     

                    It is very easy for unreserved characters to appear in the account name and/or password and possibly in the page name and any such event will stop your upload in its tracks.

                     

                    My method of dealing with this is two-fold:

                     

                    A. use the FM Filter() function to auto enter the page name on it being amended in my App so that only the unreserved characters are permitted at source within my App

                     

                    B. use the FM EncodeAsURL() function to encode the account name and password before they are entered into the CURL expression prior to calling the Applescript.

                     

                    Apologies to anyone for who feels offended by my stating the obvious here :-) but like all these things it was, for me, by no means obvious until I understood what was happening. I investigated why some automatic web site uploads worked and some didn't and could then see, using the Terminal command above, that a password of say AbC&123 was being received by the ftp server as AbC 123.

                     

                    As you have control over what you want to call your web pages it is simplest to just remove any offending characters but that option clearly doesn't apply to the ftp credentials and her his makes sense to me to use this FM function clearly intended for exactly this purpose.

                     

                    Cheers, Nick

                    • 7. Re: FTP uploads from Filemaker
                      NickLightbody

                      Here is the shorter version:

                       

                      "The best way to ensure that type of script using CURL under Applescript or Windows events works as expected is to use the Filemaker function GetAsURLEncoded () encode the inputs, then characters which are illegal in a URL will not stop your Curl command from functioning.

                       

                      A second useful item when debugging CURL is this form which returns to you what the FTP server received:

                       

                      curl --trace -n -v -ssl -l ftp://<username>:<password>@<svr IP address>/<folder>/<file.html>

                       

                      In deskspacecms, we trap for an error on upload and if one occurs we pass the output of this CURL command back to the user in a dialogue so they can work out where they went wrong."

                       

                      Cheers Nick

                      • 8. Re: FTP uploads from Filemaker
                        user19752

                        Pay attention to using password in command line, admin of the machine the command is running may be able to see it.

                        • 9. Re: FTP uploads from Filemaker
                          monkeybreadsoftware

                          and here are tips to translate from CURL in command line to CURL in MBS Plugin:

                           

                          https://www.mbsplugins.de/archive/2013-09-26/CURL_from_command_line_to_Plug

                           

                          when running inside FileMaker, it is more efficient and you have more control about what happens.

                          • 10. Re: FTP uploads from Filemaker
                            NickLightbody

                            That's either the admin of your own web server (ie you) or the admin of your own machine (ie you)? Have I got that right?

                            So if you can see your own password where is the issue - can you explain your concern a little more?

                            Thanks! Nick

                            • 11. Re: FTP uploads from Filemaker
                              NickLightbody

                              Hi Christian,

                               

                              Can you explain a little more how your plugin adds to what curl can deliver please?

                               

                              BTW - I was pleased to discover that calling curl through a Windows Event was very similar to calling it through AppleScript under OSX, just a set of single quotes difference. Does your plugin also help Windows users?

                               

                              Cheers, Nick

                              • 12. Re: FTP uploads from Filemaker
                                user19752

                                I meant admin of the os. While running the command, admin can see it using something like

                                ps Axw

                                • 13. Re: FTP uploads from Filemaker
                                  monkeybreadsoftware

                                  You can directly pass data from fields, variables or containers to the curl transfer.

                                  You can run several transfers in parallel in background or asynchronously.

                                  You don't relay on parsing the result from command line, but can query status information.

                                  You don't need to install curl extra.

                                  • 14. Re: FTP uploads from Filemaker
                                    NickLightbody

                                    Thanks Christian

                                     

                                    Nick.

                                    1 2 Previous Next