Something like you are describing would likely require a significant effrort in coding outside of filemaker (to create a plug-in etc). It is farily trivial, however, to use internet based SMS services to send and receive messages in filemaker.
The application/client does not have reliable internet access, hence 2G SMS is preferred.
Does the plugin exist?
I have previously attached a Nokia mobile phone to the serial port on a Windows PC and connected the phone to FileMaker to send/receive SMS messages, using the Troi Serial plug-in (www.troi.com).
This application was mainly to receive SMS messages from the phone - in a year we received 700,000 messages! Each message was saved in its own FileMaker record.
This was a few years ago and Troi have since changed the licensing rules for the Serial Plug-in - you now have to pay $ 999.00 for a developer license.
I believe there are other Serial Plug-ins for FileMaker available and any of these should let you connect to your USB modem. You will have to learn the codes required to send and receive SMS messages through your modem.
Best wishes - Alan Stirling - London, UK.
Thanks! I'll check out troi.
There's a few other "toolbox" plugins too that might assist you in developing some sort of interaction interface to your modem.
24U has it's toolbox that can execute system scripts: http://www.24usoftware.com/Toolbox
Monkeybread is a pretty robust toolkit: http://www.monkeybreadsoftware.de/filemaker/
360works has ScriptMaster to run Java: http://www.360works.com/scriptmaster/ (free!)
There's a plugin that does PHP as well: http://www.scodigo.com/products/smartpill-php
And lastly, there's an SMS plugin already that might interest you: http://www.fmsms.com/
There's also the "send event" or "applescript" script steps in filemaker that can execute system scripts (and are underused / extremely powerful / often misunderstood in my opinion).
For things like this, it often takes a bit of elbow grease and "outside the filemaker box" thinking to get a working solution up and running.