...with screen capture you are picking up monitor issues such as white balance and gamma settings. You are going to have a different result with EVERY monitor you use. My understanding is that PMS colors have very precisely defined RGB equivalents that data-wise will transfer properly, but you will have to adjust each monitor for white balance and gamma to get a more accurate (yet never perfect) screen representation.
Sorry I'm not offering solutions- just the reality I have worked with for over a decade now, developing data bases in the printing industry...
With the upgrade of object backgrounds in FM12, I just use a 1x1px blob of PNG image from photoshop to handle my color specific objects. This is the most accurate translation of color I've seen. While Filemaker should gracefully handle the RGB() color attribute (I have seen this in the webdirect code at least), as you've noted, it can sometimes be different. I believe this is just based on whatever renders the color. Web browsers will render different than FileMaker, and Photoshop (geared towards exact color), may be using much more precise color rendering it it's UI, in comparison to other programs.
I have only ever had one client where this really mattered. The general RGB setting has worked fine for all other times.
Also, I've seen different color behaviour on Windows vs. Mac as well, so there's that to think about.