The google maps API has a method for this. Ironically they use the bermuda triangle as an example:
This could be modified for use in FileMaker via a webviewer or potentially a page that responds in Insert From URL.
As for native functions inside of filemaker, there's not really anything setup for poly-areas.
there are plugins that make it easier as well
I've done some work on doing this using just built-in FileMaker functionality.
For smaller geographic areas, this is a basic example of the point-in-polygon problem. There are some geometric sub-functions for this like detecting when lines cross and determining if three points are oriented clockwise or counter-clockwise to each other.
When working with points on the surface of Earth, you need to clarify if the "lines" for your polygons are geodesic (great circle, shortest path) segments or instead "linear" with respect to the coordinate system. Different software uses different definitions (or both!), but the trend seems to be towards the latter, which is much more computationally convenient.
Quickly finding polygons in a database containing a point can be a very different problem. For an approach that can be adapted well to FileMaker, check out what they did for Accumulo.