New cURL options in Insert from URL

Document created by mrwatson-gbs on Apr 11, 2017Last modified by mrwatson-gbs on Oct 24, 2017
Version 10Show Document
  • View in full screen mode

Screen Shot 2017-04-11 at 16.53.36.png

cURL Specification: https://www.ietf.org/rfc/rfc3986.txt

 

 

Getting started with the new cURL options is quite daunting and a steep learning curve.

 

As there seems to be no good source about the purpose of the various FileMaker cURL options, I am starting this document to fill that gap - based on the ETS-info from FMI + my own internet research notes (a far from complete knowledge base ;-) ).

 

Note: THIS IS A WORK IN PROGRESS ! Absolutely no guarantee of completeness or correctness.

 

…but it is a starting point.

 

I am not applying any authoring restrictions ... so do please add to and improve this document and coordinate the co-authoring via the comments.

 

Hoping to be helpful, yours

 

MrWatson

 

------------------------------------------------------------------

 

Version Date: 2017-04-12

Authors: mrwatson-gbs

 

Table of Contents

 

  • New in fm16
  • What is cURL?
  • Differences between FM and the cURL command
  • Catalog of cURL options
  • How to implement old httppost: calls using cURL options

 

New in fm16

 

  • In fm16 Insert from URL now supports cURL options
    Screen Shot 2017-04-11 at 16.59.50.png
  • Most cURL options are supported - but not security-options, as this is dealt with by the FileMaker platform.
  • Note that the "httppost:" protocol has been discontinued in fm16, as this can now be done using new cURL options override old httppost: protocoll + ?-params

 

 

What is cURL?

 

"cURL is a command line tool for getting or sending files using URL syntax." (cURL - Wikipedia)

 

Basically cURL is a command which can

 

  • Fetch website pages, passing parameters where necessary encoded in the URL (i.e. using GET)
  • Send web forms to online sites, passing parameters as name/value pairs (i.e. using POST)
  • Upload data or file(s) to web sites
  • Interact with RESTful Web Services (Create/Read/Update/Delete actions - see CRUD - Wikipedia - using POST/GET/PUT/DELETE)
  • Interact with FTP servers
  • can deal with all the complexities of authorization, authentication, encryption, page forwarding and error handling which your browser does mostly in the background.
  • …and more

 

It may help to get some background knowledge in the Hypertext Transfer Protocol - Wikipedia which is the basis of much of cURLs communication.

The syntax of the URL in cURL depends on which protocol is used - see "Uniform Resource Identifier (URI): Generic Syntax": https://www.ietf.org/rfc/rfc3986.txt

 

More <TODO> (Importance of http-headers, etc.)

 

 

 

Differences between FM and the cURL command

 

  • Insert From URL only supports these protocols: http, https, ftp, ftps, & file - the cURL command supports more
  • the cURL command can read/write to files, FM-cURL can only read/write to $[$]variables (unless it is the one file referenced in the file: protocol)
    • Syntax: @$[$]myVar, example: "--data @$jsonRequest"
  • the cURL command asks for passwords, if they are empty, FM-cURL does not
  • the cURL command default is --keepalive, FM-cURL default is --no-keepalive
  • the cURL command can specify multiple URLs in one request, FM-cURL can handle a single URL
  • the cURL command cannot use the httppost: / httpposts: protocols, FM-cURL can … HOWEVER, it is deprecated and overridden.

 

None of the cURL options allow the user to access or create items on their file system. This is done for multithreading and security reasons. Instead variables are used instead as the file container. Syntax = @$[$]variable. used

 

Catalog of cURL options

 

I have tried to organise the cURL options here into semantic categories.

 

Error Handling and Debugging Options

 

Errors can be read using Get(LastError) and Get(LastExternalErrorDetail) (formerly Get(LastODBCError) )

 

Get(LastError) Returns the error number

Any InsertFromURL error that does not fall into any other defined error codes was defaulted to error number 1631.
Get(LastExternalErrorDetail) Returns the error message

All strings returned by Get(LastExternalErrorDetail) will be in English and not localized.

 

Notes:

  • Unknown options are silently ignored and cause no errors => beware of typos in your code!
  • If cURL itself returns an error that is not currently mapped, error code 1631 will be returned and Get(LastExternalErrorDetail) will contain the string that the cURL command line tool would return
  • If the call succeeds but a response code of 400 or greater is returned with the data, the appropriate error code will be returned, defaulting to 1631 if there isn't an appropriate one, and Get(LastExternalErrorDetail) will contain a string like "Response code: nnn" with nnn replaced with the response code. With show-option off, the old behavior of only mapping response code 401 to error code 1627 and all other response codes to error code 0 will happen

 

cURL Options:

 

-f / --fail (no- = DON'T)
Fail  silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal  with failed  attempts.  In  normal cases when an HTTP server fails to deliver a document, it  returns  an  HTML  document  stating  so (which  often  also describes why and more). This flag will prevent curl from outputting that and return error 22.


-S / --show-error (no- = DON'T)
Change the error handling: While the show-option is enabled, any other FileMaker error code that is generated internally, which most likely would only be an out of memory error code, will return a string like "FileMaker error: 2" with 2 being replaced by the FileMaker code. Note that this last part will only happen in rare edge cases and unlikely will be able to make a reproducible test case.


-D / --dump-header <$[$]fmvariable>
Dumps the received header information into the variable //Write the protocol headers to the specified file. This option is handy to use when you want to store  the  headers that  an  HTTP site sends to you. Cookies from the headers could then be read in a  second  curl  invocation  by  using  the  -b, --cookie  option! The -c, --cookie-jar option is a better way to store cookies. When used in FTP, the FTP server response lines  are  considered being "headers" and thus are saved there. If this option is used several times, the last one will be used.


--trace <$[$]fmvariable>Dumps the trace information into the variable (ASCII + Hex)Example: --trace $$cURL_traceDump


--trace-ascii <$[$]fmvariable>Dump trace to variable only in ASCII


--trace-time (no- = DON'T)
Prepends a time stamp to each trace or verbose line that curl displays.

 

Addressing & Networking Options

 

Options affecting the way out

 

--interface <name>

Perform  an operation using a specified interface. You can enter interface name, IP address or host name. An example  could  look like:  curl --interface eth0:1 http://www.netscape.com/ //  If this option is used several times, the last one will be used.


--socks4 <host[:port]>

// Use the specified SOCKS4 proxy. If the port number is not specified, it is assumed at port 1080.This  option overrides any previous use of -x, --proxy, as they are mutually exclusive.


--socks4a <host[:port]>

// Use the specified SOCKS4a proxy. If the port number is not specified, it is assumed at port 1080.This  option overrides any previous use of -x, --proxy, as they are mutually exclusive.


--socks5 <host[:port]>

// Use the specified SOCKS5  proxy  -  but resolve the host name locally. If the port number is not specified, it is  assumed at port 1080. This  option overrides any previous use of -x, --proxy, as they are mutually exclusive.


--socks5-hostname <host[:port]>

// Use the specified SOCKS5 proxy (and let the  proxy resolve the host  name).  If the port number is not specified, it is assumed at port 1080. This  option overrides any previous use of -x, --proxy, as they are mutually exclusive.

 

Address + path modification

 

--resolve <host:port:address>

Provide a custom address for a specific host and port pair. Using this, you can make the curl requests(s) use a specified address and prevent the otherwise normally resolved address to be used. Consider it a sort of /etc/hosts alternative provided on the command line. The port number should be the number used for the specific protocol the host will be used for. It means you need several entries if you want to provide address for the same host but different ports. This option can be used many times to add many host names to resolve.


--path-as-is (no- = DON'T)

Leave sequences of /../ or /./ in the given URL path as they are. Normally curl will squash or merge them according to standards but with this option set you tell it not to do that. (Added in 7.42.0)

 

Way in

 

--local-port <num>[-num]

individual or range of ports // Set a preferred number or range of local port numbers to use for the connection(s).  Note that port numbers by nature  are  a scarce  resource  that will be busy at times so setting this range to something too narrow might cause unnecessary connection setup failures. (Added in 7.15.2)

 

Dealing with redirection

 

-L / --location (no- = DON'T)

(HTTP/HTTPS) If the server reports that the requested  page  has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make  curl  redo  the request on the new place. If used together with -i, --include or -I, --head, headers from all requested pages will be shown. When authentication  is  used, curl only sends its credentials to the initial host. If a redirect takes curl to a different  host,  it won't  be  able to intercept the user+password. See also --loca- tion-trusted on how to change this. You can limit the amount  of redirects to follow by using the --max-redirs option. When  curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl  will  re-send  the  following request using the same unmodified method. You  can  tell  curl to not change the non-GET request method to GET after a 30x response by  using  the  dedicated  options  for that: --post301, --post302 and -post303.


--location-trusted (no- = DON'T)

(HTTP/HTTPS) Like -L/--location, but will allow sending the name + password to all hosts that the site may redirect to. This may or may not introduce a security breach if the site redirects you do a site to which you’ll send your authentication info (which is plaintext in the case of HTTP Basic authentication).

--post301 (no- = DON'T)

(HTTP) Tells curl to respect RFC 2616/10.3.2 and not convert POST requests into GET requests when following a 301 redirection. The non-RFC behaviour is ubiquitous in web browsers, so curl does the conversion by default to maintain consistency. However, a server may require a POST to remain a POST after such a redirection. This option is meaningful only when using -L, --location (Added in 7.17.1)
--post302 (no- = DON'T)
(HTTP) Tells curl to respect RFC 2616/10.3.2 and not convert POST requests into GET requests when following a 302 redirection. The non-RFC behaviour is ubiquitous in web browsers, so curl does the conversion by default to maintain consistency. However, a server may require a POST to remain a POST after such a redirection. This option is meaningful only when using -L, --location (Added in 7.17.1)
--post303 (no- = DON'T)

(HTTP) Tells curl to respect RFC 2616/10.3.2 and not convert POST requests into GET requests when following a 303 redirection. The non-RFC behaviour is ubiquitous in web browsers, so curl does the conversion by default to maintain consistency. However, a server may require a POST to remain a POST after such a redirection. This option is meaningful only when using -L, --location (Added in 7.17.1)

Proxy

 

(ToDo: What is a proxy?)

 

--noproxy <host[,host] or *>

//


--proxy-header <name: value>

// use this option multiple times to set multiple header entries


-p / --proxytunnel (no- = DON'T)


--proxy-anyauth (no- = DON'T)


--proxy-basic (no- = DON'T)


--proxy-digest (no- = DON'T)


--proxy1.0 <proxyhost[:port]>

 

//
-U / --proxy-user <user:password>

Unlike the command line tool, if the password does not exist we do not ask the user for it


-x / --proxy <[protocol://] [user:password@]proxyhost[:port]> // ...

 

Connection + transfer

 

--tr-encoding (no- = DON'T)

(HTTP) Request a compressed Transfer-Encoding response using one of the algorithms curl supports, and uncompress the data while receiving it. (Added in 7.21.6)


--raw (no- = DON'T)

(HTTP) Disable all internal HTTP decoding of content or transfer encodings and instead makes them passed on unaltered, raw. (Added in 7.16.2)

Time & speed

 

-y / --speed-time <time>

integer value of seconds


-Y / --speed-limit <speed>

integer value of bytes per second


--limit-rate <speed[b|B|k|K|m|M|g|G>

integer followed by optional suffix


-z / --time-cond <date expression>

Only supports specifying a cURL style date expression // (HTTP/FTP) Request a file that has been modified later than  the given  time  and date, or one that has been modified before that time. The <date expression> can be all sorts of date strings  or if it doesn't match any internal ones, it is taken as a filename and tries to get  the  modification  date  (mtime)  from  <file> instead.  See  the curl_getdate(3) man pages for date expression details.
Start the date expression with a dash ( - ) to make it request for a  document that is older than the given date/time, default is a document that is newer than the specified date/time.
If this option is used several times, the last one will be used.


--connect-timeout <seconds>

Decimal number of seconds


--expect100-timeout <seconds>

Decimal number of seconds


--keepalive (no- = DON'T)

Enable the use of keepalive messages on the TCP connection. By default FM-cURL disables them (which is the OPPOSITE behaviour of the curl command)


--keepalive-time <seconds>

integer number of seconds


-R / --remote-time (no- = DON'T)


-m / --max-time <seconds>

decimal number of seconds


--tcp-nodelay (no- = DON'T)

Turn on the TCP_NODELAY option. See the curl_easy_setopt(3) man page for details about this option. (Added in 7.11.2)

 

Command & Header Options

 

-X / --request <command>

Change the (HTTP) Command


-I / --head (no- = DON'T)

(HTTP/FTP/FILE) Fetch the HTTP-header only! HTTP-servers feature the  command  HEAD which this uses to get nothing but the header of a document. When used on an FTP or FILE file,  curl  displays the file size and last modification time only.


-H / --header <name: value>Use this option multiple times to set multiple header entries.

  • See https://en.wikipedia.org/wiki/List_of_HTTP_header_fields for general info about http headers
  • Content negotiation headers
    • Accept // Content-Types that are acceptable for the response. See Content negotiation. // Accept: text/plain or application/javascript
    • Accept-Charset // Character sets that are acceptable // Accept-Charset: utf-8
    • Accept-Encoding // List of acceptable encodings. See HTTP compression. // Accept-Encoding: gzip, deflate
    • Accept-Language // List of acceptable human languages for response. See Content negotiation. // Accept-Language: en-US
    • Accept-Datetime // Acceptable version in time Accept-Datetime: Thu, 31 May 2007 20:35:00 GMT (Provisional)
    • Expect // Indicates that particular server behaviors are required by the client // Expect: 100-continue
  • User-Agent // what browser
  • Range // see  --range
  • Referer // see --referer
  • Pragma // Implementation-specific fields that may have various effects anywhere along the request-response chain. // Pragma: no-cache

 

-e / --referer <URL>

(?) the previous web page - does support ";auto" at end of url when used with --location

 

 

Authentication & Identification

 

--anyauth (no- = DON'T)

Use any authorisation (--no-anyauth does nothing)

 

--digest (no- = DON'T)

(HTTP) Enables HTTP Digest authentication. This is an authentication scheme that prevents the password from being sent over the wire in clear text. Use this in combination with the normal -u, --user option to set user name and password. See also --ntlm, --negotiate and --anyauth for related options. If this option is used several times, only the first one is used.

 

-u / --user <user:password>

Unlike the command line tool, if the password does not exist we do not ask the user for it

 

-A / --user-agent <agent string>

S. Name/Version (Kommentar)

 

 

Cookies / Session

 

 

-b / --cookie <name=data[;n2=d2]>

Set a cookie

 

-b / --cookie <$[$]fmvariable>

Use cookie data from FM variable

 

-b / --cookie-jar <$[$]fmvariable>

Direct file access replaced with FM variable

 

-j / --junk-session-cookies (no- = DON'T)

 

Encryption (of Communication)

 

--ssl //

--ssl-allow-beast (no- = DON'T) // also --ftp-ssl

--ssl-reqd (no- = DON'T)

--ciphers <list of ciphers>

(SSL) Specifies which ciphers to use in the connection. The list of ciphers must specify valid ciphers. Read up on SSL cipher list details on this URL: https://www.openssl.org/docs/apps/ciphers.html // NSS ciphers are done differently than OpenSSL and GnuTLS. The full list of NSS ciphers is in the NSSCipherSuite entry at this URL: https://git.fedora-hosted.org/cgit/mod_nss.git/plain/docs/mod_nss.html#Directives // If this option is used several times, the last one will be used.

 

Encoding (of Data)

 

--FM-text-encoding <encoding>

FileMaker specific option, for any option following this one the specified encoding will be used when converting FileMaker's internal text format for use by the cURL engine

-B / --use-ascii (no- = DON'T) // Use ASCII

-- / --crlf (no- = DON'T) // Use CRLF

 

Data Options

 

Form data (name=value pairs)

 

-F / --form <name=content> // form data (quoted?)

-F / --form <name=@$[$]fmvariable> // form data value from variable

--form-string <name=content> // form data (quoted?)

 

Data (single string, structured JSON, file content)

 

-d / --data <data>

Data (quoted)

 

-d / --data @<$[$]fmvariable>

Data from fm variable (@ character prefixing a FM variable replaces direct file access, overrides httppost:)

 

--data-ascii <data>

ASCII data (quoted)

 

--data-ascii @<$[$]fmvariable>

ASCII data from fm variable

 

--data-binary <data>

Binary data (quoted)

 

--data-binary @<$[$]fmvariable>

Binary data from fm variable

 

--data-raw <data>

data (raw (???))

 

--data-urlencode <data>

Data (quoted)

 

--data-urlencode =<data>

Data (quoted??)

 

--data-urlencode <name>=<data>

Data (quoted??)

 

--data-urlencode @<$[$]fmvariable>

Data from fm variable

 

--data-urlencode <name>=@<$[$]fmvariable>

Data from file from fm variable path (@ character prefixing a FM variable replaces direct file access, overrides httppost:)

 

File upload/download

 

-T / --upload-file <$[$]fmvariable>

Direct file access replaced with FM variable, no globbing support, file name in URL

 

-o / --output <filename>

Set the file name for container field data (download)

 

Getting/Putting a substring of data

 

--ignore-content-length (no- = DON'T)

 

--max-filesize <bytes>

note = BYTES not CHARACTERS!

 

-r / --range <range>

(HTTP/FTP/SFTP/FILE) Retrieve a byte range (i.e a partial document) from a HTTP/1.1, FTP or SFTP server or a local FILE. Ranges can be specified in a number of ways.

0-499 specifies the first 500 bytes

500-999 specifies the second 500 bytes

-500 specifies the last 500 bytes

9500- specifies the bytes from offset 9500 and forward

0-0,-1 specifies the first and last byte only(*)(H)

500-700,600-799 specifies 300 bytes from offset 500(H)

100-199,500-599 specifies two separate 100-byte ranges(*)(H) (*) = NOTE that this will cause the server to reply with a multipart response!

Only digit characters (0-9) are valid in the 'start' and 'stop' fields of the 'start-stop' range syntax. If a non-digit character is given in the range, the server's response will be unspecified, depending on the server's configuration.

You should also be aware that many HTTP/1.1 servers do not have this feature enabled, so that when you attempt to get a range, you'll instead get the whole document.

FTP and SFTP range downloads only support the simple 'start-stop' syntax (optionally with one of the numbers omitted). FTP use depends on the extended FTP command SIZE.

If this option is used several times, the last one will be used.

 

 

 

 

-C / --continue-at <offset>

"-" for current file size is not supported. Data replaces, not appends, to binary targets

 

FTP Options

 

FTP Encryption and Adressing

 

--ftp-ssl-reqd (no- = DON'T)

 

--ftp-ssl-ccc (no- = DON'T)

Also sets ftp-ssl-ccc-mode to passive if it is not already set

 

--ftp-ssl-ccc-mode <active or passive>

Specify one of two modes

 

--ftp-ssl-control (no- = DON'T)

 

-P / --ftp-port <interface or IP or host or ->

Also can append ":[start]-[end]" to specify port range (what does - mean?)

 

FTP Authentification

 

--ftp-account <data>

 

--ftp-alternative-to-user <command>

 

FTP Settings

 

--ftp-pasv //

--ftp-skip-pasv-ip (no- = DON'T)

 

--disable-eprt (or --no-eprt)

(FTP) Tell curl to disable the use of the EPRT and LPRT commands when doing active FTP transfers. Curl will normally always first attempt to use EPRT, then LPRT before using PORT, but with this option, it will use PORT right away. EPRT and LPRT are exten- sions to the original FTP protocol, and may not work on all servers, but they enable more functionality in a better way than the traditional PORT command. --eprt can be used to explicitly enable EPRT again and --no-eprt is an alias for --disable-eprt. Disabling EPRT only changes the active behavior. If you want to switch to passive mode you need to not use -P, --ftp-port or force it with --ftp-pasv.

 

--disable-epsv (or --no-epsv)

(FTP) Tell curl to disable the use of the EPSV command when doing passive FTP transfers. Curl will normally always first attempt to use EPSV before PASV, but with this option, it will not try using EPSV. --epsv can be used to explicitly enable EPSV again and --no-epsv is an alias for --disable-epsv.

Disabling EPSV only changes the passive behavior. If you want to switch to active mode you need to use -P, --ftp-port.

 

 

FTP Command

 

--list-only

(FTP) When listing an FTP directory, this switch forces a name-only view. This is especially useful if the user wants to machine-parse the contents of an FTP directory since the normal directory view doesn't use a standard look or format. When used like this, the option causes a NLST command to be sent to the server instead of LIST. Note: Some FTP servers list only files in their response to NLST; they do not include sub-directories and symbolic links. (POP3) When retrieving a specific email from POP3, this switch forces a LIST command to be performed instead of RETR. This is particularly useful if the user wants to see if a specific message id exists on the server and what size it is. Note: When combined with -X, --request <command>, this option can be used to send an UIDL command instead, so the user may use the email's unique identifier rather than it's message id to make the request. (Added in 7.21.5)

 

--ftp-create-dirs (no- = DON'T)

 

--ftp-method <multicwd, nocwd, or singlecwd>

Specify one of three methods

 

--ftp-pret (no- = DON'T)

 

-Q / --quote <command>

use this option multiple times to send multiple commands to ftp server

 

IMAP-Server Options

 

Don't know if this is relevant ... or quite how it works …I think this is only required when FileMaker is functioning as an IMAP server ....

 

--sasl-ir (no- = DON'T)

SASL initial response (is supported)

See rfc document "IMAP Extension for Simple Authentication and Security Layer (SASL) - Initial Client Response" https://tools.ietf.org/html/rfc4959

Notes:

 

 

 

 

How to implement old httppost: calls using cURL options

 

<TODO>

5 people found this helpful

Attachments

    Outcomes