Some apps have url schemes so you can enter that in your open url request button and it will open the app instead of a web browser. But that is not going to get your native FileMaker form. ( as far as I know ).
Have you looked at any of the FM specific drawing packages? FileMaker Drawing Tool - GoDraw v2.0 - Draw Arrows, Lines and Shapes or FMEasyCanvas | An Open Source Drawing Solution for the FileMaker Platform
i Sent touch draw an email asking if they had a url scheme as I use it as well.
I didn't know about those FM drawing options. I haven't downloaded them yet. When you have the drawing, is it possible to export the drawing as vector data to be used in AI or ArcGIS? It doesn't need to be geo-tagged in anyway. I need to include a drawing with each form, the drawing then gets entered into our final reports for my company. We usually take paper drawings, scan, trace in AI and then turn them into PDFs.
We're trying to get the drawing ability to open up right from the form. I know it can be done- saw it at a conference a few months ago and several people had done this.