First, do you have a backup of that file? I'd go back to that. Because using "Force Quit" FileMaker to stop a running database is not good for it, and may damage the file eventually if you do it often; though scripts are pretty strong; much better than the machine going down while you're changing a Field's calculation, or a Relationship!
As far as stopping a script (like a never ending Loop), I find that Cmd - Period ( i.e., . ) seems to work; a lot better than Cmd - ~, or [esc] ("escape").
If the FileMaker application is hung it won't respond to any commands. If the backup script normally executes properly the problem seems to be the app is frozen. Recently I made some changes and when my closer script would run, it hung on a step it couldn't evaluate. My fault. However command/period wouldn't stop the script. Even the debugger hung. The only solution was to force quit. Hosted file so less risky, but still no fun.