Wondering if anyone else has tested this function and found an issue.
I filed a bug report but I'd love to be proven wrong
What you found seems like a great thing I would code as a unit test. That is the inverse of the function on a value should be the value again, itself.
Based on all the bugs that are popping up, assuming you have found a bug too, which it sounds like, it doesn't seem that testing is very rigorous at FMI.
regardless of anything, G(F(X)) = X must hold every time F() is the inverse of G().
I'm not sure what FileMaker anticipated this function to be used for.
If you do the same unit test with an hex entirely representing visible characters it works
something like HexEncode( HexDecode( "46696C654D616B6572" ) ) works
but that's limited for the real world
I suspect that the issue is that you are passing around FileMaker text which might be sanitized for your protection.
HexEncode (HexDecode ( "969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d"; "thefile.bin" ))
to make it use container objects instead.
I had tested that and you are correct. However, using the extra parameters makes things even worse once you use the HexDecode inside CryptAuthCode:
CryptAuthCode( "data" ; "SHA256" ; HexDecode ( "969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d"; "thefile.bin" ))
HexDecode passes the string "thefile.bin" as the decoded key
That may indicate a bug since the Example 2 in the documentation for CryptAuthCode() has that structure suggesting that it should work.
The third argument to CryptAuthCode() is "any expression or field" which does not strongly imply to me that a Container is acceptable there.
let's see what comes of the bug report. this conversation will be good supporting material
How about using Base64Decode( Base64Encode(container)) to get the key data out?
CryptAuthCode( "data" ; "SHA256" ; Base64Decode(Base64Encode(HexDecode ( "969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d"; "thefile.bin" ))))
not as far as i can see.
as stated in the doc the "key" parameter in in CryptAuthCode can be "any expression or field" but the one we are submitting has (and needs to have) non-printable unicode characters when decoded, no matters the format we decode from.
my feeling is that the problem lays there in FileMaker relying on a limited set of text characters when decoding.
Yes... Thank you!
A simple unit test, right?
(The developers do test this stuff, right? I am starting to wonder...)
For example, Sin(ArcSin(30)) Had better be.... 30.
HexEncode is not for converting decimal into hexadecimal and vice versa HexDecode is not for converting hexadecimal into decimal. The example you've given is suggesting that you think it does.
If I take your example in the bug report and reverse the function it works fine:
HexDecode ( HexEncode( "969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d" ) )
as a result
HexEncode( "969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d" )
If a function for decoding a hexadecimal number into a decimal number would be used on your string the result should be:
You can do that with little CF which I got from WinfriedHuslik (whom unfortunately has passed away last year):
Let ( [ h = "0123456789ABCDEF" ; hex = Filter ( Upper ( hex ) ; h ) ; d = Position ( h ; Left ( hex ; 1 ) ; 1 ; 1 ) - 1 ; s = Length ( hex ) - 1 ] ; GetAsNumber ( d * ( 16 ^ s ) + Case ( s > 0 ; CF_hex2dec ( Right ( hex ; s ) ) ) ))
Let ( [
h = "0123456789ABCDEF" ;
hex = Filter ( Upper ( hex ) ; h ) ;
d = Position ( h ; Left ( hex ; 1 ) ; 1 ; 1 ) - 1 ;
s = Length ( hex ) - 1
GetAsNumber ( d * ( 16 ^ s ) + Case ( s > 0 ; CF_hex2dec ( Right ( hex ; s ) ) ) )
Thanks Menno ! that and other CF's are out there for DEC->HEX->DEC.
for others following this thread, a good study of the Container Functions:
FileMaker Pro 16 Help
Plus you will start seeing a lot more articles and tips at DevCon for ways these can be used.
I was previously only reading the documentation with no hands-on experience.
After studying the HexEncode() and HexDecode() directly with FileMaker 16, it appears that when you use text arguments to these functions (instead of containers) work via UTF-8. They are not an arbitrary binary coder/decoder functions when working with FileMaker Text since Text is not a raw binary type.
HexEncode("FileMaker") will first interpret the string "FileMaker" as UTF-8 (no BOM) and return the bytes as hexadecimal.
That string consists entirely of ASCII 7-bit characters so there is only one byte per character resulting in
"46696C654D616B6572" as the output.
HexEncode("€") // The Euro character Char(8364)
is represented in UTF-8 (no BOM) by 3 bytes.
The resulting output is "E282AC".
HexDecode(some hex sequence) will attempt to interpret that hex sequence as proper UTF-8 (no BOM) string.
Instead of returning 3 binary values
HexDecode("E282AC") will return the Euro character "€" since that is how it is expressed in UTF-8.
HexDecode(an arbitrary Hex string of a binary sequence)
will not work since that string of bytes will be interpreted as UTF-8 and will likely fail to be valid.
If you need to work with raw binary, use the Container versions of these functions.
HexDecode("E282AC"; "myfile.bin") will generate a container holding the file containing those 3 bytes.
HexDecode("FFFEAC20"; "myfile.bin") will generate a container holding a file containing those 4 bytes. That just happens to be the same Euro symbol expressed in UTF-16LE (with BOM).
In the case of the Key argument to CryptAuthCode(), I don't think there is a way to pass a raw binary key. It looks like it wants FileMaker Text.
Good assessment and thanks for testing, Tom!
See TextEncode() for variations on encoding.
Interesting! Nice job.
siplus wrote: regardless of anything, G(F(X)) = X must hold every time F() is the inverse of G().
The situation here is that F(X) is being applied when X is not a valid argument to F(X). (Think F(x) = 1/x and x is 0.)
In this case it appears that HexDecode(hexString) is only valid for values of hexString that represent valid UTF-8 strings. As long as that is the case,
HexEncode(HexDecode(hexString)) = hexString
HexDecode(HexEncode(inputText)) = inputText
is true for all values of inputText since inputText is always representable as UTF-8 (despite being stored as UTF-16LE internally in FileMaker).
TomHays wrote: The situation here is that F(X) is being applied when X is not a valid argument to F(X). (Think F(x) = 1/x and x is 0.)
good point !
Thanks for the time put in the detailed explanation Menno.
I know Winfried CFs but they don't apply to my problem: the AWS 4 signature doesn't use decimal numbers, it uses strings of non printable characters.
that said I don't want to "reverse" the functions: the key is not arbitrary, it is the result of a previous HexEncode.
Hence my expectation HexDecode will give me back the original Hexed text
Thank you Tom. Great analysis!
Not being able to use CryptAuthCode to calculate the HMAC of the biggest provider of web services in the world still feels quite limited to me, especially when the documentation doesn't mention these limitations.
will stick to the BaseElement plugin for now.
for the complete process (which might help understanding why I'm using that key) look here Examples of How to Derive a Signing Key for Signature Version 4 - Amazon Web Services.
The test keys are at the bottom of the page. The first step uses a plain text key, like Tom said, and it return a correct value for kDate. Everything breaks after that because of the limitations described above.
I was trying to replace the BE plugin to have a local FM Go file I could use with AWS.
Looking at this post, the replies, though good, seem all over the place.
Are you trying to do a simple Base64Encode or are you trying to do HMAC AWS crypto stuff?
It would be helpful (for me, at least) if you would post a string example of what you want to encode, how you want to encode it, how you plan to use it, and your expected/desired result.
Let's make sure we can understand exactly what you want first.
True. I've had a reply about the bug report and clarified the process there. Here's the same explanation if you're interested
I had come to the conclusion about UTF-8 with my tests but I think it's good to clarify what I was trying to do:
the process to sign an Amazon Web Services request requires to generate some Hex keys and they are non UTF-8 on purpose. The process in FileMaker using the BaseElements plugin looks like
kDate = BE_HMAC ( dateStamp ; "AWS4"&key ; BE_MessageDigestAlgorithm_SHA256 ; BE_Encoding_Hex ) ;
kRegion = BE_HMAC ( StringToHex( regionName ); kDate; BE_MessageDigestAlgorithm_SHA256 ; BE_Encoding_Hex ; BE_Encoding_Hex ) ;
with the key generated at each step containing non-UTF8 characters
The new function CryptAuthCode is supposed to do HMAC in a similar way, but the following doesn't return the same result as above
kDate = HexEncode( CryptAuthCode ( dateStamp ; "SHA256" ; "AWS4"&key ));
kRegion = HexEncode(CryptAuthCode ( HexEncode( regionName ) ; "SHA256"; HexDecode( kDate ; "data.bin" ) ) ) ;
the test values provided by Amazon are
key = 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'dateStamp = '20120215'regionName = 'us-east-1'
and the intermediate results for testing are
kDate = '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d'kRegion = '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c'
What I would do in this case, just to see what's going on, is to use another language to generate the cryptographic hashes to compare to (1) What Amazon expects, and (2) What FMP Generates.
I have some code I use in Java, but from what you've posted above, I can't map your example to it.
My "basic" code uses a secret and a message. And returns a hash which is base64 encoded using an HMAC.
So, for any possible comparison for you, if your secret key is "secret", and your message is "Message", that code returns this hash:
Thank you for your posts.
One of our developers noticed this forum thread and confirms that CryptAuthCode function returns a different result than Base Elements. An issue has been filed by Development.
Any way to figure out which is right?
IOW, is there a "known-good" combination that generates an agreed-to result?
My apologies for not making this clearer.
Base Elements has the correct result. The issue is with FileMaker's CryptAuthCode function when Hex encoding option is used. When more information becomes available, I will post again.
user16087 (et al):
This issue has been addressed in FileMaker Pro 16.0.2.
Thanks FileMaker for the quick turnaround on this
Retrieving data ...